You aren't particularly doing anything
wrong, you just weren't ready
yet. As soon as you start killing Core Guard Posts on the AI Homeworld, that Homeworld starts sending those Exowaves at you. So, you need to be prepared to completely wipe the targeted AI Homeworld before the reaction Exowaves can kill you. One the AI Home Command Station on that homeworld is destroyed, the Exowaves will stop.
Your options to speed up your homeworld assaults generally fall into three categories:
1) Get more Knowledge to unlock more ships
2) Build Warheads
3) Capture Fabricators, Advanced Factories, Golems, asteroids, or any other useful things scattered around the galaxy
4) Use Hacking to improve your unit effectiveness
In your game, you haven't captured either of the Spire Archives yet (9000 K each).
You have 280 Hacking Progress left to spend.
You have several good Fabricators you haven't captured or hacked - Specifically the Bomber Mk V, Decoy Drones, Zenith Devastator, and Spire Starship fabs all produce excellent units.
For Hacking on the AI Homeworld, Sabotage works well - while it won't kill the Guardposts, it will kill stuff like the Mk III Fortress.
It also looks like your AI Homeworld assault started from Fenyunock, right next to the AI Home Command Station. This means you were immediately subjected to the attacks of the Mk III Fortress and the Reserves while they were still covered by the Mk V Forcefields. Coming in through Zaunde (with all or part of your fleet) may have helped kill off the Shredder Guardpost that killed a bunch of your ships. I don't know what else there was in that system, or where it was located: what you did may have been best.
As for using Warheads, using a part of your fleet to trigger the spawning of the Reserves, then using a few Lightning or Armored Warheads to kill them all off without losses is popular tactic, especially when the AI Home Command Station is close to an entry wormhole.
On the defensive front, you have very few defenses, and they are not positioned to help you fight off Exowaves. More turrets, on more planets, placed to force the AI to fight past them on the way to your Homeworld, will give you more time to rebuild your fleet and conduct follow-up attacks once your first try has ended.
Kahuna has an
excellent guide on how to build defenses (and other good advice) - I strongly recommend studying it.
So, while I think your game as attached is pretty much hopeless, if you have a save from before the Homeworld assault, you are in good shape to win. You've got plenty of AIP headroom, so get some more K, do some more hacking, build up some defenses, and build up your fleet. Then try again.