I think a combination of skies quality and optical performances drive the rest of the story, assuming you get decent tracking figures.
For a given focal length, there are not many ways to widen the possibilities :
focal reducers are probably the most innocent solution, providing shorter f and faster F/D, which are interesting for deep sky larger targets.
barlow may increase reach, but at the expense of f/d (longer exposures needed) and unchanged separation power (linked to d)
So each instrument has its own set of limitation , or should I say its own set of capabilities to be optimistic, but there is definitely no “one size fits all” approach, especially for deep sky where targets size range is tremendous, from larger nebulae (eg Spaghetti) or smaller planetary nebulae.
I don’t think playing with camera sensor size is a good path, for several reasons :
unless you want to cover extreme opposite targets (high speed solar vs deep sky for instance) , multiplying camera is expensive and rather inefficient, because it doesn’t dramatically change optical constraints of a given instrument
as of today, APS-C cooled camera are the sweet spot solution for deep sky imaging; moreover, it is possible to tackle with binning to address smaller target and longer-f instruments, should the pixels be too small
As regards instrument, I would (and I did it, btw) acquire something with a focal length close to the limit of available seeing conditions. In my opinion, this is in the range 1000-1200mm. Refractor vs reflector is a mater of personal choice, I’ve no religion about it. For refractor, the sweet spot is around 120mm for d, to my opinion.
For that reason, I have (too, according to my wife) many instruments, from camera lenses up to 180mm, then four scopes (350, 550, 900, 1200) and a RC8 @1600mm. For deep sky imaging , I only use APS-C format.