After commenting I had a theory and it may be right. I have dual WAN for redundancy and setup a routing policy for ml and world to route through my WAN that is not CGNAT, going with the assumption that CGNAT sometimes the public IP is blocked. The primary problem with it is that images will break when federating and after doing this it seems to be working better.

That being said it all started happening after I used the Lemmy state checker and I assume since it queries the endpoint for the selected site on an interval I got flagged by something.