General principle: do as much as you can without removing and remounting the part.
Make sure the planes of the XY table and the rotary table are perpendicular to the Z axis and the spindle. Play around using a large diameter fly cutter if you thing they are already aligned. You might be surprised.
If you use a tool plate you can fasten your part with screws from underneath and leave it fastened until it is done. You'll also be able to drill and mill all the way through the part (as long as you don't run into your fixture screws. Sherline sells a round tool plate for the rotary table, or you can buy some MIC-6 and make your own.
If you mill the center hole by rotating the rotary table, you'll be guaranteed it is concentric with the table and subsequent features.
You may want to fly cut the material if you are using 6061.
Go for CNC for sure, but you can do a lot with a rotary table on a mill.