Using nukes makes it harder by raising the AI Progress, so they should be a desperation move.
What you may need to do is gather some more knowledge to unlock more/better ships, as those will help you take the AI out. Science ships can do that in systems you already have (up to 3000/system, the map can show you how much you've gotten from a system), or you can use a knowledge hacker ship to get knowledge from AI worlds without having to take them (this makes the AI angry, but doesn't raise AI Progress).