You can't do a good job of painting or varnishing a hull, rudder or spars if the fittings are in the way, so best to remove them.
If you are unfamiliar with your boat’s fittings, it may be worth taking photos or notes of where they were before you start. Keep the fixings for a fitting with it, so when it comes to re-fitting it later, you have the correct size bolts, washers, nuts or screws to hand. If you are going to wash your fittings (always a good idea if you have time), just leave them in the hole in the fitting.
I generally remove all the fittings except the drainage hole fittings (mine are glued in, rather than screwed), inspection hatch surrounds on the foreward bulkhead, side decks or side tank side panels and glued on (wooden) mast steps.
I would strongly recommend removing the self bailer since water often gathers around it when the boat is not being sailed, and also when it is. So any failure of the paint/varnish around the hole the bailer sits in is likely to lead to water getting into the wood, resulting in paint/varnish lifting, damage to the plywood or rot.
