Maybe long refleet times are caused by multiple expensive bonus ships and possibly somewhat careless use of the ships? "Yarrr I have a big fleet so I'll just blob around". Lose 20% of your fleet every time you do that and it starts to add up.
That's probably part of it, for sure. I tend to be fairly cautious with the fleet overall, but not particularly cognizant of ship vs. ship matchups, so I might lose a large portion of some expensive unit, while losing little of everything else.
(When we got guardians, I recall this being particularly devastating. "Oh, it's got a medium hull, fighters then! OH GOD IT'S EATING MY FIGHTERS, WTF. Oh, it does AOE attacks...it's supposed to be countered by frigates, despite having a bonus against artillery....")