Yeah, I go back and forth on Minecraft. Open-ended infinite world is fine, but you really do have to set yourself a goal to make it fun. When I decided I was going to build a nether portal, for example, I had a good few hours of fun getting down to bedrock and finding diamonds for a pick and then a lava pool and a bucket of water to make obsidian blocks to mine out and so on. But then once I had it all built and decorated around it with trees and such to make it look suitably spooky and nether-like, I was again suddenly bored. That's why I'm saying that with no goals, you have no game.
At the most basic level, in order to have a game you have to have a goal to reach and you have to have something opposing you in reaching that goal. Everything from chess to lacrosse to TF2 follows this basic pattern. I win if I do X, and the other person or the developer-placed obstacles try and stop me. That's a game. You may be setting your own goals, or you may be following side goals like achievements. But you can't just set out to say 'I'm not going to attempt to do anything' and then get a game out of it. You win already, game over!