Romhacking was a lot of fun, I was one of the earlier ones in that scene.
The DLL is not sandboxed in the slightest, it has full access to the extent that your user account does (basically like any other program). That said, depending on your OS, if it requires rights elevation for you to do certain activities, then those would also apply to the game. So for instance in windows, even if you are an administrator on your computer, certain things require that you run a program in "Run As Administrator" mode to do, otherwise they give access denied messages.
THAT said, there's a lot of stuff that's completely unprotected. There's not a thing in the world from one of our games, or anybody else's game for that matter, from going in and deleting every one of the files in your documents folder (on your respective OS), for instance. We can't delete your OS, because that requires rights elevation, and trying to upload documents would probably fail because of your firewall unless you gave it permission (which actually you usually do because of the updater checks), but there's nothing to stop rampant deletions of actual data.
Scary? Sure, I suppose so. But in that same light, there is absolutely nothing in the world stopping the people you invite into your home from stabbing you to death as soon as you look away. They could steal from under your nose, hold you at gunpoint, whatever. I mean, if we're talking sandboxing and permissions, they are not sandboxed away from doing any of those things.
The reason you trust game devs and other software vendors not to do that sort of malware is pretty much the same reason you trust the guys from a cleaning service or an insurance service or whatever to come into your home. There's a professional relationship there, and if the guy who installs your gutters starts stabbing his clients, well, that's the end of the line for him in his business, as well as hopefully jail. Same deal with people who make software for your computer: if we're malicious about it, we'll be found out, and that leads both to the end of the business and probably to criminal charges of some sort.
Personally, I like to think that people make the right choices not because they are afraid of getting caught, but because they have an inherent moral compass that is pointed the right way. I've had people do a lot of terrible things to me and people I know in my life, so I know that's not always the case, but in 99% of people I think it's probably true.
When it comes to anonymous mod authors, or romhackers for that matter, there's less of a risk of them getting caught, and less to lose. But I think it's fairly straightforward to tell the ones who are super excited about making a mod or a romhack for its own sake, and those who might be... questionable. Let's be honest: if someone wants to have some serious mischief with your computer, there are a lot easier ways than making an elaborate mod for a niche game.
That said, there's also nothing stopping some third party from hosting the mod, injecting it with malware, and then passing it on as if nothing was changed. THAT's one of the easier ways to get onto your computer. I again think it's really unlikely, but basically it's a good argument for only downloading from the modder's original links, and not mirrors that someone else has put up. Not that people who make mirrors are bad, either -- they are also often some of the most passionate, helpful people. So, as with all things in life, it's a judgement call.
Wow I'm rambling. It's doing me good to think about something else other than my main work right at the moment. I've been sitting here having panic attacks for the last couple of hours (I'm not entirely sure why), and this took my mind off things in a good way. I like extrapolating all the possibilities that exist in a situation like this, and security was always an area that I found very interesting and take very seriously. Anyhow, now that that's done, back to work for me...