If you are the player, I believe the ideal number is two.
Then, based on my spreadsheet, the collective efficiently gradually drops until it hits the 0.06 cap at around 8 ships, where it bottoms out. Then it starts gradually creeping back up again until it reaches the same total damage output as 2 ships at 20 ships. So, 2 sentinel frigates will do the same as 20, in other words. Then, beyond 20, the gain is always a net positive, so as long as the AI has 2 or fewer or 20 or more, they are in an ideal zone. But, the shots of the sentinel frigates are individually weaker when they have something like 65 ships, which makes it so they are not instantly killing your ships too easily -- nice for you as the player, but also reasonably effective for the AI.
For them, having 65 ships is the equivalent of firepower of about 3.9 sentinel frigates with no penalty (so, the attack power of a single sentinel on one planet x 3.9).
Regarding the rounding errors, those are just fixed-int-math precision errors, not much I can do about those. People complained about those with the autocannons, too. It does make them a hair less predictable if you are trying to spreadsheet them, but they are consistently applied in the game so that works well enough for purposes of the game. The basic issue is that for these float-type math operations, it only has 3 decimal places of precision because of the non-floating-point number type I have to use for both speed and cross-CPU consistency with the game. That low of a precision does not cause issues often, but with this sort of math it does lead to mildly wrong (but consistent across CPUs, again) results.