upgrading to a 4080 or 4090 wouldn't make enough of a difference to be justifiable.
Agreed. I just went from a 3060 to a 4080 and performance was better but only took BXT times for an APS-C from around 30 sec to around 20 or so. Not worth it for that. I did it mainly for gaming performance, however. The 4000 series may also may require a bigger PSU, depending on what you have. PCPartPicker can do an estimate for you.
If I did not care about gaming, I would get a 3060 or 3070 and maybe even wait until the 5000 series comes out and get a 3000 series. They are now in the mid 300 range and would come down even more.