RAW/FITS Compatibility Issues with ASIAIR - Forums

Topic starter

Mar 27, 2023

Hi folks:

I've just switched from Astroberry to ASIAIR for image acquisition. I use a Canon 600D/T3i, and previously all downloaded images were saved in RAW (CR2) format. I have an extensive darks library all saved in RAW format. When I stack, Astro Pixel Processor creates master darks in FITS format, and they calibrate my light frames, also RAW, without issue.

Jump forward to switching to the ASIAIR. It only saves downloaded images in FITS format. The dimensions no longer match the RAW dark files in my library, nor does it match the FITS master darks created from those RAW files. I even have a few nights of flats I took without turning on the ASIAIR that I fear I won't be able to use. The dimensions are completely different:

FITS files from ASIAIR: 5,202 x 3,465 px RGGB
FITS master darks from CR2 dark frames: 5,344 x 3,516 px GBRG

Of course the sensor specs say 5,184 x 3,456 but I'm guessing that comes down to how it's debayered or something. I've read accounts of folks converting the CR2 files to FITS and cropping, but how would you ensure that the dark and light frames match, and how do I change the bayer matrix so they're compatible? Should I try re-saving the FITS darks as RGGB, aligning the dead pixels and cropping to fit the lights, and then applying the same steps to the flats?

I realize I should be ditching the darks library and shooting entirely new darks, but it's starting to warm up here and it'll be difficult for me to match tempertures. The flats are impossible to re-shoot, sadly, as I've changed filters several times. Moving forward I'll know to shoot all my calibration frames through the ASIAIR, but is there anything I can do to create useable calibration frames from my RAW files?

CS,
Mark

Well Written Insightful Respectful Engaging

Michael Gorman

Mar 27, 2023 · Edited: Mar 27, 2023

it'll be difficult for me to match tempertures

Wouldn't that mean all your new lights also can't hit the low temps? You want the temp to match on the darks and the lights. I would as i imagine most would, recommend just making new ones. I can't see any benefit in spending all that time trying to convert the old ones.

Concise

Mark Germani

Topic starter

Mar 27, 2023

Michael Gorman:
it'll be difficult for me to match tempertures

Wouldn't that mean all your new lights also can't hit the low temps? You want the temp to match on the darks and the lights. I would as i imagine most would, recommend just making new ones. I can't see any benefit in spending all that time trying to convert the old ones.

Hi Michael, thanks for the response. I already have the light frames, I just need darks to match them. I always take my darks on cloudy nights, as we get so few clear nights here on the West Coast of Canada, I can't leave my gear unattended overnight, and I can't stay up with it as I have a job and a kid to get to school in the morning. I can reshoot some of the darks (and did so last night) but others taken in January and February were at temps well below current night-time temps, and I have a few nights worth of flats that I really want to be able to use.

Joe Linington

Mar 27, 2023 · Edited: Mar 27, 2023

With my DSLR's I used darks that were close enough (within 3-5c either way). It worked well. I have even used fridge darks for images taken much colder, like -15c and it has been decent. Living in BC, your fridge is a perfect 3-4c which should be close ish. Give it a shot and you may be surprised how well it calibrates. The other option is to find a friend with a wine fridge. I have a wine fridge that is adjustable between 32f and 60f and have used that to cool the camera down. I don't know what to do for anything in the -2 to -10 range (not that that happens often there). Most freezers are below -10.

I ran into the same Raw vs Fits issue when I switched to NINA. They just didn't want to work together with slight differences in pixel count. I reshot my library in NINA and just used NINA for everything.

Helpful

Michael Gorman

Mar 27, 2023

Mark Germani:
Michael Gorman:
it'll be difficult for me to match tempertures

Wouldn't that mean all your new lights also can't hit the low temps? You want the temp to match on the darks and the lights. I would as i imagine most would, recommend just making new ones. I can't see any benefit in spending all that time trying to convert the old ones.

Hi Michael, thanks for the response. I already have the light frames, I just need darks to match them. I always take my darks on cloudy nights, as we get so few clear nights here on the West Coast of Canada, I can't leave my gear unattended overnight, and I can't stay up with it as I have a job and a kid to get to school in the morning. I can reshoot some of the darks (and did so last night) but others taken in January and February were at temps well below current night-time temps, and I have a few nights worth of flats that I really want to be able to use.

Gotcha,

I hear you on all those points. I'm in Idaho here in the US so not terribly far from you. Also have a kid and real life responsiblities and I'm in the process of building an Shed + Dome observatory so i can leave my stuff setup to get the most data i can on any clear spot of any night.

The temp stuff is why I moved from photo camera to Cooler equipped astro camera fairly quick in my progression. That way i can basically have a single master Dark for months to a year at a time with dead on temps. So worth the investment.

Good luck trying to get sort your situation.

Supportive

Pleiotrope

Mar 27, 2023

I don’t know if this might help you find a solution, but I have seen the ASIAIR encode row order in FITS files differently than Ekos did. Since your Canon is really a RGGB Bayer matrix, the fact that APP is using GBRG to debayer may suggest that the row order between the ASIAIR FITS and CR2 files (when converted to FITS) is different.

It may be- though I’m not sure how exactly- that this results in some pixels being discarded by one debayer process but kept by another, resulting in different file sizes. There’s a discussion here that might be relevant.

Maybe you could try forcing different debayer patterns on either your lights or when re-creating a master dark and see if that affects file dimensions?

Well Written Helpful Insightful Respectful Engaging Supportive

Mark Germani

Topic starter

Mar 27, 2023 · Edited: Mar 27, 2023

Joe Linington:
With my DSLR's I used darks that were close enough (within 3-5c either way). It worked well. I have even used fridge darks for images taken much colder, like -15c and it has been decent. Living in BC, your fridge is a perfect 3-4c which should be close ish.

Thanks Joe! I try and stay within 2 C, but yeah - I've done fridge darks before. I usually leave the camera in the fridge 30 minutes to let it cool a bit first - same when I do cloudy-night darks. I suspect I'm going to be doing a lot of rebuilding, and I combine previous years' sessions with new sessions all the time, so I'll have to keep two separate darks libraries - one for RAW and one for FITS!

I have to laugh at how overly-complicated this hobby is at times. I was telling somebody earlier today that there is such a contrast between the amazing images we're able to create, and the tedious process. Of course, those of us who are hooked tend to enjoy most if not all of the process, but it's still amusing.

Returning to the issue at hand, I can't reshoot my flats from sessions earlier this year so I'd still love to know if there's anything I can do to convert my RAW files to match the FITS...

Well Written Insightful Respectful Engaging

Joe Linington

Mar 27, 2023

Mark Germani:
I have to laugh at how overly-complicated this hobby is at times. I was telling somebody earlier today that there is such a contrast between the amazing images we're able to create, and the tedious process. Of course, those of us who are hooked tend to enjoy most if not all of the process, but it's still amusing...

I'm transitioning to a 294m cooled camera and my first project took 270 calibration frames (30 flats and dark flats for each LRGB of different times plus 30 Darks) to calibrate 130 lights. Crazy ridiculous.

Mark Germani

Topic starter

Mar 27, 2023

Maybe you could try forcing different debayer patterns on either your lights or when re-creating a master dark and see if that affects file dimensions?

Thanks @Pleiotrope, I'll see if I can play with the debayer settings in APP - I don't think I can get the ASIAIR to alter its settings in that regard - to see if I can get them to align. Looks like the ASIAIR is using the same bayer matrix as my camera, but APP is converting it when creating the stacked masters. It's always managed to work out in the past in APP while stacking, so it must work it out during the calibration...

Mark Germani

Topic starter

Mar 27, 2023

Michael Gorman:
The temp stuff is why I moved from photo camera to Cooler equipped astro camera fairly quick in my progression.

Thanks Michael - yeah, I had hoped to move to a cooled camera (got my eye on an ASI533MC) but funds were not available

I ended up buying PixInsight, which has been great as I've found that processing may have been the weak point in my images thus far. Can't wait to start processing some of these targets I've imaged over the past few months, but gotta figure out this RAW/FITS issue!

CS,
Mark

Oscar

Mar 27, 2023 · Edited: Mar 27, 2023

Mark Germani:
I ended up buying PixInsight, which has been great as I've found that processing may have been the weak point in my images thus far. Can't wait to start processing some of these targets I've imaged over the past few months

Yep, there's nothing that PI can't do. I'm amazed at how much better images look (mine included) when they are processed in PI.
And it's not so hard to master either, it just takes some time.

Gary JONES

Mar 27, 2023

Hi Mark,
I just spotted your thread and thought this info might help …

Many camera sensors (particularly in DSLRS) have a few rows/columns of pixels around the edges that the camera uses for calibration.

Files generated by the camera don't add these rows/columns to the image, so your RAW files are probably 5,184 x 3,456.

My guess is that the ASIAir is using the entire sensor = 5,202 (18 extra columns) x 3,465 (9 extra rows).

I'm not sure why your Master Darks are different, but the switching of the Bayer pattern doesnt look right.

IMHO the only way is to reshoot your calibration frames, particularly your flats as you mention you have changed filters.

Well Written Helpful Insightful Concise

Mark Germani

Topic starter

Mar 27, 2023

Gary JONES:
Many camera sensors (particularly in DSLRS) have a few rows/columns of pixels around the edges that the camera uses for calibration.

Hi Gary, thanks! I think I see a bit of that in some of the calibration frames. I have reshot my darks, but I think I'll also have to redo the flats and just clone-stamp the dust where it doesn't match the new flats. Lesson learned, I guess!

Well Written Respectful

Gary JONES

Mar 28, 2023

Ah yes - this is not a simple hobby - the more I learn, the more I realise I have no idea …

I'm not entirely certain what you mean re your flats …
Are you proposing to use new flats with old images, or old flats with new images ?

The thing with flats is that they're usually different for every session - thanks to old dust leaving and new dust arriving,
and if you swap filters, they will be different for each filter.

Yesterday I shot 20 light flats and 20 dark flats for each of 7 filters - 280 flats all together - not for the faint-hearted

But I've set up a workflow to automate it, so I just turn the power on and click 'go' and I get a new set of 280 flats in about 20 minutes.

The only way to get the best result is to shoot flats for every session, they are pretty quick to do once you get the hang of it

I usually do these after my rig has settled and stabilised temperature, while I'm waiting for twilight to end.

BTW - best to do your light flats after you have achieved focus - changing focus will change the size of the dust blobs

Helpful Engaging

Mark Germani

Topic starter

Mar 28, 2023 · Edited: Mar 28, 2023

Gary JONES:
I'm not entirely certain what you mean re your flats ...
Are you proposing to use new flats with old images, or old flats with new images ?

Hi Gary. The problem is that I took the flats already, and they are in RAW. I took them directly after the sessions, but not using the ASIAIR. They don't match the lights' dimensions, and I have since changed filters (which for me involves removing the camera) several times. I won't be able to use these flats, but I know if I retake them they won't line up as well as flats taken directly after the session. Unless I can figure out a way to make the dimensions and bayer matrix match, my only option is to retake them, though, as the vignetting will be similar, at least, and the mote of dust on my sensor will be in the same place, and any other random dust I'll have to clone-stamp out. I'm just running out of options for the sessions where I took the flats in RAW and acquired images using the ASIAIR.

Gary JONES

Mar 28, 2023 · Edited: Mar 28, 2023

Ah - OK - I see your dilemma.

Here's what I'd try ...
Use Photoshop or Affinity etc and create 2 layers @ 5,202 x 3,465.

Place an OLD flat from the camera (5,184 x 3,456) onto one layer, and a NEW light from AISAir (5,202 x 3,465) on the other layer.

Shift the Canon layer to align the dust spots - you should find that the layers differ by 18 columns and 9 rows.

Create a 5,184 x 3,456 mask that matches the Canon layer, and superimpose that over the ASIAir layer.

Then merge down the 2 layers (the mask and the ASIAir layer) so you have an image @ 5,184 x 3,456 then save that as your new flat.

Repeat for each flat - then your flats should be the same as your lights.

If you use Affinity Photo (IMHO the best app - same as PhotoShop but includes astrophotography stacking etc), you can save these files directly as FITS.
You can get free trial of Affinity Photo here - it's cross-platform, really cheap and you don't have to renew the license every year like you do with Adobe.

Good luck - maybe post back here and let us all know how things turn out

Gary

Helpful Engaging Supportive

Mark Germani

Topic starter

Mar 28, 2023 · Edited: Mar 28, 2023

Gary JONES:
Here's what I'd try ...

Ok - I'll try this out when I get home, and see how far it gets me. Fingers crossed; many thanks!

Well Written Respectful

Ron Kramer

Mar 31, 2023

Mark Germani:
Hi folks:

I've just switched from Astroberry to ASIAIR for image acquisition. I use a Canon 600D/T3i, and previously all downloaded images were saved in RAW (CR2) format. I have an extensive darks library all saved in RAW format. When I stack, Astro Pixel Processor creates master darks in FITS format, and they calibrate my light frames, also RAW, without issue.

Jump forward to switching to the ASIAIR. It only saves downloaded images in FITS format. The dimensions no longer match the RAW dark files in my library, nor does it match the FITS master darks created from those RAW files. I even have a few nights of flats I took without turning on the ASIAIR that I fear I won't be able to use. The dimensions are completely different:

FITS files from ASIAIR: 5,202 x 3,465 px RGGB
FITS master darks from CR2 dark frames: 5,344 x 3,516 px GBRG

Of course the sensor specs say 5,184 x 3,456 but I'm guessing that comes down to how it's debayered or something. I've read accounts of folks converting the CR2 files to FITS and cropping, but how would you ensure that the dark and light frames match, and how do I change the bayer matrix so they're compatible? Should I try re-saving the FITS darks as RGGB, aligning the dead pixels and cropping to fit the lights, and then applying the same steps to the flats?

I realize I should be ditching the darks library and shooting entirely new darks, but it's starting to warm up here and it'll be difficult for me to match tempertures. The flats are impossible to re-shoot, sadly, as I've changed filters several times. Moving forward I'll know to shoot all my calibration frames through the ASIAIR, but is there anything I can do to create useable calibration frames from my RAW files?

CS,
Mark

There is a easy answer. Don't use land based cameras for astro. Pick up a "COOLED" astrocam. Also stack with astropixel processor. It stacks anything with anything. I dropped pixinsight 3 years ago after using it for 2 years. Best thing I ever did in this hobby. I also stopped using calibration frames. No darks, flats or superdarks or other B.S. You should work in fit. But then I did a stack for someone online and APP loaded this raw files just as I load .fit files. I didn't try - but I BET it will stack fit and raw as it stacks all combinations of files. I stacked from 4 different resolution cameras that were on 3 different scopes over the course of 4 years... mono and OSC subs. It stacked them all with no issues.
M78 https://www.flickr.com/photos/portranet/50856162833/in/dateposted/

Gary JONES

Apr 1, 2023

Ron Kramer:
Mark Germani:
Hi folks:

I've just switched from Astroberry to ASIAIR for image acquisition. I use a Canon 600D/T3i, and previously all downloaded images were saved in RAW (CR2) format. I have an extensive darks library all saved in RAW format. When I stack, Astro Pixel Processor creates master darks in FITS format, and they calibrate my light frames, also RAW, without issue.

Jump forward to switching to the ASIAIR. It only saves downloaded images in FITS format. The dimensions no longer match the RAW dark files in my library, nor does it match the FITS master darks created from those RAW files. I even have a few nights of flats I took without turning on the ASIAIR that I fear I won't be able to use. The dimensions are completely different:

FITS files from ASIAIR: 5,202 x 3,465 px RGGB
FITS master darks from CR2 dark frames: 5,344 x 3,516 px GBRG

Of course the sensor specs say 5,184 x 3,456 but I'm guessing that comes down to how it's debayered or something. I've read accounts of folks converting the CR2 files to FITS and cropping, but how would you ensure that the dark and light frames match, and how do I change the bayer matrix so they're compatible? Should I try re-saving the FITS darks as RGGB, aligning the dead pixels and cropping to fit the lights, and then applying the same steps to the flats?

I realize I should be ditching the darks library and shooting entirely new darks, but it's starting to warm up here and it'll be difficult for me to match tempertures. The flats are impossible to re-shoot, sadly, as I've changed filters several times. Moving forward I'll know to shoot all my calibration frames through the ASIAIR, but is there anything I can do to create useable calibration frames from my RAW files?

CS,
Mark

There is a easy answer. Don't use land based cameras for astro. Pick up a "COOLED" astrocam. Also stack with astropixel processor. It stacks anything with anything. I dropped pixinsight 3 years ago after using it for 2 years. Best thing I ever did in this hobby. I also stopped using calibration frames. No darks, flats or superdarks or other B.S. You should work in fit. But then I did a stack for someone online and APP loaded this raw files just as I load .fit files. I didn't try - but I BET it will stack fit and raw as it stacks all combinations of files. I stacked from 4 different resolution cameras that were on 3 different scopes over the course of 4 years... mono and OSC subs. It stacked them all with no issues.
M78 https://www.flickr.com/photos/portranet/50856162833/in/dateposted/

Yes - I'd expect any stacking program to be able to stack images from different cameras - provided the images contain stars, and are more-or-less centred on the same target.

The original issue was about calibration frames - which don't contain stars - so one wouldn't ordinarily expect APP or any other program to be able to align calibration frames correctly with lights, unless they have identical resolutions.

Well Written Helpful Insightful Concise