Does the 533mm Pro pair well with Samyang 135 and 3nm narrowband filters?

This is basically the same as the 2600mm and I have had and used both with this lens. I have used both 3nm and 4.5nm filters as well as broadband. Something to consider about 3nm Ha is the exclusion of Hbeta. Some NB examples:


Not done with the Samyang, but the Nikkor AI 135mm F/2.8 using the 533MM, 4.5nm Ha & 3nm Oiii/Sii: https://www.astrobin.com/nvkfa6

ASI533MM, Samyang 135mm, 4.5nm Ha - This was purely a backfocus testing image i made: https://www.astrobin.com/1ds9r7/0/

ASI533MM, Samyang 135mm, 4.5nm Ha, 3nm Oiii: https://www.astrobin.com/rnsy4e/

ASI2600MM, Samyang 135mm, 3nm Oiii/Ha: https://www.astrobin.com/p5xcmy/M/

3nm filters will work just fine, but I would perhaps consider 4.5nm to include Hbeta as well. If you intend to shoot at F/2 you would be better off with ones made for fast optics.
Not getting an entire field of round stars, and slightly off nebulosity like seen in my test image, is very common even on smaller sensors. This is purely down to tilt/backfocus which is crucial at those F-Ratios, each fraction of backfocus matters a lot. So spend some time adding 0.1mm spacers to the image train in order to evaluate the stars. An easier option would be to stop down slightly, which perhaps is necessary on certain copies due to poor QC. 

What I found to work for me was to try and get the backfocus as best as possible so that the stars were workable, then I just spend a few minutes re-shooting them slightly stopped down to F/2.8 or F/4. I've got some step down rings for that purpose now.

Jan Erik Vallestad:
This is basically the same as the 2600mm and I have had and used both with this lens. I have used both 3nm and 4.5nm filters as well as broadband. Something to consider about 3nm Ha is the exclusion of Hbeta

There is no such a thing. The Balmer sequence is Ha - Hb - Hg... and the visbile light frequencies of the series are:

656.3
486.1
434.0
410.2
397.0

So maybe you are referring to an OIII filter but that needs to be much wider then 4.5nm to include both the two emission lines of OIII and Hb.

I'll point out the obvious here … you will lose a great deal of potential field of view with the 533.  I'm not sure what your other options for sensors are but one of the beauties of the 135mm lenses is the expansive FOV .. which you'll only be able to capture with the 533 via mosaic

Nearly 25 square degree of sky, seems pretty extensive to me...

I wouldn’t recommend this combination. The Samyang is very wide for deep sky imaging, but the 533 is small sensor. For a wide view you want a wide sensor. A full frame or APSC size chip would be a much better pairing. The field of view of your image will be cropped 2.7x compared to a full frame sensor, which that lens supports.