Arcen Games

Other => Website Support => Topic started by: Dominus Arbitrationis on May 17, 2015, 12:53:39 PM

Title: Site Maintenance
Post by: Dominus Arbitrationis on May 17, 2015, 12:53:39 PM
I am going to be doing a bunch of stuff on the server today, so the site may go down for periods of time, but it will be back up by 8:00 AM EST tomorrow. Also, any Mantis reports made today may be missing after this, so please keep a copy of every report you submit today in the event it does disappear.

This maintenance has been rescheduled for the the 23rd and 24th of May. There is a small chance the site will be down here and there, but I do believe I can mitigate most, if not all, of it to only an hour or two of downtime total. The bolded advice above still stands true, however. While I will endeavor to save all the reports, some may slip between my fingers. :)

I believe you will be able to view the forums, but you might be unable to while I set it in maintenance mode. The wiki will be read-only until the maintenance is complete. However, you will be able to browse the wiki at all times (Most likely).

In the event you need to contact Arcen during the downtime, please send us an email at [email protected]


As of, 6:40 PM EST, 24 May, the site is officially considered as being stable. I did just have to reboot the server, however, so you may have experienced momentary downtime just now. We do now have SSL enabled, so you can browse most of arcengames.com with a secure connection. 

Thank you for your patience regarding this matter, and many apologies for any inconvenience.
Title: Re: Site Maintenance
Post by: x4000 on May 17, 2015, 12:57:48 PM
I, for one, am super excited about the transition to the new and faster site.  And I gave Quinn the go-ahead to break as much stuff today as he needs to in order to get us transitioned over quickly.  Doing a transition without breaking anything is way harder in my experience than doing a transition where you're allowed hours and hours of downtime, so I said that as long as we're back by 8am EST tomorrow morning (the 18th), I'll be happy.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 17, 2015, 10:15:19 PM
This is being postponed until next weekend, at which point I can dedicate a lot more time to this. The infrastructure was really wacky for some reason (No fault of the hosts, only the OS), but I'm pretty sure it is mostly sorted out now.

I will be working on the server this entire week when I am able to, so if someone needs me shoot me an email and I'll respond a lot faster than a forum/mantis/wiki post/issue.
Title: Re: Site Maintenance
Post by: x4000 on May 17, 2015, 11:25:39 PM
Thanks again for all your hard work on this!
Title: Re: Site Maintenance
Post by: iozay on May 23, 2015, 06:39:16 AM
Good luck during the migration, I have done this a few times myself and things will break - But after some time it really pays back :)

Good luck during these stressful times! :)
Title: Re: Site Maintenance
Post by: x4000 on May 23, 2015, 07:26:52 PM
Things are still migrating, of course, but the speed on this is already freaking incredible. :)   Happy dance bigtime for me, heh.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 23, 2015, 08:16:45 PM
I'd also like to point out I haven't even enabled a lot of the speed boosting stuff. :)

I just removed the forums from maintenance mode so everyone can post now, but there is the chance that the forums will horribly die as a result. For some reason there are odd fragments of the code appearing in the top of some of the pages and the error logs are going crazy with just about everything. But it seems usable so I want to see what happens.

Mantis currently has uploads disabled due to a issue with importing old attachments, which would make us choose between either old attachments or new ones. And no one wants that, so I'm going to keep working on fixing the old attachments then letting us post new ones.

Wordpress should be up and running perfectly.

The wiki is currently still down, but I just haven't gotten over it yet. It will probably be the last thing I do since comparatively few people use it and it holds less critical information.
Title: Re: Site Maintenance
Post by: x4000 on May 23, 2015, 08:23:20 PM
It's definitely great progress!
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 23, 2015, 09:38:29 PM
Alright, a small update regarding the forums:

I did notice the code fragments appearing at the top of certain pages and have fixed it for a few pages. If people could give me a list of all pages affected, that would be very helpful. Mantis might be best for that, but here wouldn't be terrible either.
Title: Re: Site Maintenance
Post by: Captain Jack on May 23, 2015, 11:45:58 PM
My Messages/PM has it. That's the only place I've seen it though.
Title: Re: Site Maintenance
Post by: jerith on May 24, 2015, 03:15:09 AM
We seem to have lost the ability to upload files to Mantis -- see http://www.arcengames.com/mantisbt/view.php?id=16816 for an example.

Thanks for the work on this. Moving infrastructure that's been around for a while is never easy, and I don't envy you the job. (I've done my fair share of sysadmin work and have some idea of what's involved. You, sir, are a hero.)
Title: Re: Site Maintenance
Post by: Captain Jack on May 24, 2015, 03:42:41 AM
Did you check the sticky? (http://www.arcengames.com/forums/index.php/topic,17546.0.html)  :P
Title: Re: Site Maintenance
Post by: iozay on May 24, 2015, 04:50:07 AM
Things tend to run quite smoothly so far :) Great job! :)
Title: Re: Site Maintenance
Post by: Pumpkin on May 24, 2015, 07:25:03 AM
The wiki (this address: http://arcengames.com/mediawiki/index.php?title=Main_Page (http://arcengames.com/mediawiki/index.php?title=Main_Page)) returned that.

Quote from: Wiki
Warning: require(/var/www/html/mediawiki/includes/objectcache/ObjectCache.php): failed to open stream: No such file or directory in /var/www/html/mediawiki/includes/AutoLoader.php on line 1009

Fatal error: require(): Failed opening required '/var/www/html/mediawiki/includes/objectcache/ObjectCache.php' (include_path='/var/www/html/mediawiki:/var/www/html/mediawiki/includes:/var/www/html/mediawiki/languages:.:/usr/share/pear:/usr/share/php') in /var/www/html/mediawiki/includes/AutoLoader.php on line 1009

Also, on Mantis, issues/suggestions that used to be marked "considering" with a cyan background are now marked "@[email protected]" and have a white background.

I hope this report help. Should I post it on Mantis? :P
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 24, 2015, 07:49:50 AM
Also, on Mantis, issues/suggestions that used to be marked "considering" with a cyan background are now marked "@[email protected]" and have a white background.

Fixed!
Title: Re: Site Maintenance
Post by: iozay on May 24, 2015, 10:10:29 AM
Just changed my mail adres. The following PHP code appeared on the confirmation URL:
Code: [Select]
name']) > 25) $context['checked_username'] = $smcFunc['htmltrim']($smcFunc['substr']($context['checked_username'], 0, 25)); // Only these characters are permitted. if (preg_match('~[<>&"\'=\\\]~', preg_replace('~&#(?:\\d{1,7}|x[0-9a-fA-F]{1,6});~', '', $context['checked_username'])) != 0 || $context['checked_username'] == '_' || $context['checked_username'] == '|' || strpos($context['checked_username'], '[code') !== false || strpos($context['checked_username'], '[/code') !== false) $context['valid_username'] = false; if (stristr($context['checked_username'], $txt['guest_title']) !== false) $context['valid_username'] = false; if (trim($context['checked_username']) == '') $context['valid_username'] = false; else { require_once($sourcedir . '/Subs-Members.php'); $context['valid_username'] &= isReservedName($context['checked_username'], 0, false, false) ? 0 : 1; } } ?>
The link that was used:
http://www.arcengames.com/forums/index.php?action=activate;u=13258;code=e326953d26
Title: Re: Site Maintenance
Post by: Aklyon on May 24, 2015, 10:21:12 AM
The wiki appears to be completely empty at the moment. I guess its not done with maintenance?
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 24, 2015, 10:36:11 AM
Alright, I figured it out. The problem arises from a faulty backup service that duplicated the code and put it at the end of some files. Sometime today I will probably be be refreshing the site files with unbroken ones, which may cause some downtime here.

The wiki appears to be completely empty at the moment. I guess its not done with maintenance?

Yep, the text table is refusing to export fully. So I'm trying to get everything to export properly then import it without damaging the wiki.
Title: Re: Site Maintenance
Post by: x4000 on May 24, 2015, 12:54:38 PM
What about doing just an entire database copy?  Can we do that while still updating to the newer MySQL version?  And yeah, I'm pretty pissed with codeguard.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 24, 2015, 01:00:44 PM
The database is actually so large that it can't be normally exported. Even a mysqldump takes over fifteen minutes, which is Rackspace's cutoff point. I'm trying to use the codeguard backup right now and it takes a ton of time to download then to extract from the zip. I've looked into doing replication, but Rackspace doesn't support that.
Title: Re: Site Maintenance
Post by: x4000 on May 24, 2015, 01:05:25 PM
That's for the wiki database?  :o

I know that mantis is huge with all the file BLOBs, but I thought the wiki was just text in the database and then files externally.  And very few files on that anyway, to my knowledge.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 24, 2015, 01:07:51 PM
The wiki text table is about 2.0 GiB and Mantis's file table is about 4.3GiB. The rest of the tables are pretty small and already imported. Speaking of importing, I am now about to try putting in the Mantis Files table.
Title: Re: Site Maintenance
Post by: x4000 on May 24, 2015, 01:10:20 PM
Dear god that's a lot of text.  I guess that's because of all the revision histories for massive pages.  If there's a way for us to blow out revision histories and just keep the most recent version of everything that is there right now, I'd be cool with that.  After 5+ years it's plenty time to do that anyhow.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 24, 2015, 01:16:58 PM
Nope. Revisions are stored in their own little table that is only 11 MiB. It is really scary to think about just how much text is on that wiki.

Also, sorry about the tiny bit of downtime there. The SWAP partition got stuck and I had to reboot the system otherwise nothing runs quickly at all.
Title: Re: Site Maintenance
Post by: x4000 on May 24, 2015, 01:20:18 PM
No problem at all.
Title: Re: Site Maintenance
Post by: Aklyon on May 24, 2015, 01:23:53 PM
The wiki is that big, even without the revision text? Woah.
Title: Re: Site Maintenance
Post by: x4000 on May 24, 2015, 01:38:03 PM
I have a feeling that the revisions table is a lookup table, and the text from the revisions is in the main text table, but I could be wrong.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 24, 2015, 01:43:19 PM
Nope. It stores the edit summaries in blobs in the revisions table. Of course, the blobs are really small, with the largest only being a KB or two I believe. There is a max character limit for the summaries which is what makes it so small.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 24, 2015, 02:52:14 PM
Can I have some volunteers look through Mantis and find all of the uploaded files that are missing? I believe I fixed the database so every uploaded file should be there, but I want to be sure.

EDIT: I just checked the latest issues and I do believe they are all set up. However, about 10 issues have disappeared. I will be sending the people who lost their issues emails containing the details of their post.
Title: Re: Site Maintenance
Post by: iozay on May 25, 2015, 07:59:29 AM
Why oh why did I thought that this was a mess before to start... Obviously I was right.

@Chris: You made for sure the right call at allowing some downtime, even an enterprise level organization would have scheduled at the very least a period of time to make parts of the system read only for end-users.

Good luck with ironing out the issues! :)
Title: Re: Site Maintenance
Post by: Pumpkin on May 25, 2015, 09:40:51 AM
The wiki text table is about 2.0 GiB and Mantis's file table is about 4.3GiB.
I love seeing the innards of all this. We could do some funny facts lists, like the total number of suggestions on Mantis, the number of character in the Wiki, the number of posts in the Forum...

Quote from: The Forum
156,525 Posts in 10,301 Topics by 18,007 Members.
Holly... forum!


And yeah: good luck. Seems like you need it.
Title: Re: Site Maintenance
Post by: Mánagarmr on May 25, 2015, 10:31:18 AM
Is it just me or is the page significantly slower these days?
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 10:48:07 AM
Good luck with ironing out the issues! :)

All issues should be ironed out now! The only thing left is performance-enhancing stuff and a little bit of infrastructure that you guys really won't notice.

I love seeing the innards of all this. We could do some funny facts lists, like the total number of suggestions on Mantis, the number of character in the Wiki, the number of posts in the Forum...

There are currently about 16835 reports in Mantis, the number of characters on the Wiki is impossible to store, it actually overflows the max integer value, and you found the number of posts in the forum! If you want more innards of the wiki you can look at Special:Statistics (https://arcengames.com/mediawiki/index.php?title=Special:Statistics) which is automatically updated a ton. Make sure to sign in to view the most recent stuff though! For Mantis you can see the date when each issue is resolved/closed at the Change Log (https://www.arcengames.com/mantisbt/changelog_page.php). :)

Is it just me or is the page significantly slower these days?

Its slower for you? Hmm... that might actually be a location thing then. The server is in Michigan, USA rather than spread out across the world. Although, in theory, Cloudflare should be giving you really good results still. I'll definitely look into speeding it up for you though!
Title: Re: Site Maintenance
Post by: Draco18s on May 25, 2015, 11:02:54 AM
There are currently about 16835 reports in Mantis, the number of characters on the Wiki is impossible to store, it actually overflows the max integer value, and you found the number of posts in the forum!

Number of characters = [number of gigabytes of text data] * [size of a gigabyte in bytes]

;)
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 11:05:47 AM
Number of characters = [number of gigabytes of text data] * [size of a gigabyte in bytes]

;)

Heh, that would work. Wouldn't be perfect, but relatively close. There are other tables and other bits of data in each table that add to the size :P
Title: Re: Site Maintenance
Post by: Draco18s on May 25, 2015, 11:14:22 AM
Number of characters = [number of gigabytes of text data] * [size of a gigabyte in bytes]

;)

Heh, that would work. Wouldn't be perfect, but relatively close. There are other tables and other bits of data in each table that add to the size :P

It's a good ballpark estimate and its not like the exact count is a useful number in any meaningful fashion anyway.  "Oh, I'm sorry, the number we were looking for was seventeen qaudrillion, four hundred and fifteen trillion, seventy five billion, two hundred and thirty nine million, seven thousand, and one."
Title: Re: Site Maintenance
Post by: x4000 on May 25, 2015, 11:18:15 AM
Mantis just went down for about 30 seconds, and the wiki is still acting incredibly slow right now.  Is this maintenance or...?
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 11:19:55 AM
That was an Apache restart to update the configuration stuff. Tell me if it stays slow though.
Title: Re: Site Maintenance
Post by: x4000 on May 25, 2015, 12:15:42 PM
Sorry, have been in mantis like crazy the past while and forgot to check back.  The wiki is still super slow.  It's on 60+ seconds and counting for one page refresh for me right now, although the prior one was 4 seconds.  Table locking contention, maybe?
Title: Re: Site Maintenance
Post by: x4000 on May 25, 2015, 12:16:15 PM
And then after hitting cancel and then refreshing again, it refreshed in 1 second.  So that's all over the place.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 12:47:37 PM
The only thing I can think of to cause that would be table locking contention, but we don't have any MyISAM tables that would lock the whole table, only row-level locking is available. It might be a client thing, I know that my browser occasionally takes forever to load a page at one point but then loads the rest of that site extremely fast. When you were doing the page refresh, did you have ?action= in the URL?
Title: Re: Site Maintenance
Post by: Mánagarmr on May 25, 2015, 01:14:35 PM
I'm in Sweden and loading a reply page takes ca 15 seconds. It never used to be that slow.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 01:33:00 PM
I'm in Sweden and loading a reply page takes ca 15 seconds. It never used to be that slow.

That is definitely unacceptable for the speed. Does it only happen on the forums, on certain forum actions, or is it site-wide? Also, what actions are you taking when it is slow?
Title: Re: Site Maintenance
Post by: Mánagarmr on May 25, 2015, 01:35:24 PM
I'm in Sweden and loading a reply page takes ca 15 seconds. It never used to be that slow.

That is definitely unacceptable for the speed. Does it only happen on the forums, on certain forum actions, or is it site-wide? Also, what actions are you taking when it is slow?
It seems to be rather intermittent. I've had it happen twice now today, but right NOW when I replied to you both loading the forum and the reply page was snappy.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 01:37:41 PM
That suggests it is more of a database problem. I'll look into it and see if I can diagnose the speed issues. When it does happen, try to refresh and bypass the cache (The shortcut depends on your browser).
Title: Re: Site Maintenance
Post by: x4000 on May 25, 2015, 02:10:05 PM
It's entirely possible that my thing was a browser thing, I'm not sure.  I did not have action in the title, I had http://arcengames.com/mediawiki/index.php?title=Stars_Beyond_Reach_Beta_Release_Notes#Version_0.841 as the url I was using.

It looks like every other time I refresh it, it goes superslow in my browser.  But if I hit enter to just "go to that url," then it works fine and snappy.  Thanks chrome!!  It's strange, though, because other sites don't do that for me.

On the old host, however, even just going there fresh was not working with any speed, so this is a marked improvement in almost every way.  Just some lingering browser stupidity apparently.

edit: performance to Sweden aside. :/
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 04:02:42 PM
The only way to be sure would be to try other browsers. If it isn't just a browser issue then I'm concerned about what is causing random slowdowns.

That sounds like your browser has the old server's pages cached and the new server is telling it "No, that is really outdated, accept my copy." and chrome is not wanting to update its cache.

Yea, the new server is definitely performing really well. It is odd that Sweden would be affected negatively, given I believe the server is physically closer, and Cloudflare should mitigate the whole distance problem.
Title: Re: Site Maintenance
Post by: x4000 on May 25, 2015, 04:03:30 PM
I'm not sure how much caching cloudflare is actually doing at the moment based on its configuration?
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 04:09:00 PM
It should be caching everything. Which may actually be to blame for bad connections, since the cached data is not at all related to the new data at this point.
Title: Re: Site Maintenance
Post by: x4000 on May 25, 2015, 04:12:14 PM
It's only doing that much caching if you changed a lot of settings (which feel free to).  There is a way to force flush the cache I believe, as well.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 04:22:20 PM
I think you're thinking of the security setting, which is highly reduced. The cache is set to cache everything except the wiki, which the wiki handles its own caching (Or will, once I get around to that). I found the clear cache button, which will hopefully make it faster for everyone and reduce the browser errors you were getting.
Title: Re: Site Maintenance
Post by: Radiant Phoenix on May 25, 2015, 05:13:38 PM
I'm getting occasional 521 errors, but it usually goes away within a minute.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 05:14:37 PM
521s are caused by the services restarting in response to something, so nothing to worry about. :)
Title: Re: Site Maintenance
Post by: jerith on May 25, 2015, 06:31:13 PM
I see occasional hanging connections when I refresh the wiki, specifically http://arcengames.com/mediawiki/index.php?title=Stars_Beyond_Reach_Beta_Release_Notes which is particularly interesting to me at present. This happened before the migration as well, with about the same frequency, but I don't remember it being a problem during the TLF beta which was the last time I hammered the wiki like this.

I don't always get the hanging connections, but when I do it's pretty consistently on one or two of these resources:

Without looking at the network tab of Chrome's developer tools, it just looks like a very slow page load and might be related to what Chris was seeing. If I hit cmd+shift+R to bypass caching/last-modified (thanks for the hint about caching), I get a pretty snappy load. If I hit cmd+R for a normal reload, I get the hanging connections more often than not. Those look like scripts, CSS, and images, which shouldn't be changing at all between page loads.

Hope this helps you debug this. I'm going to just use cmd+shift+R on the wiki from now on. ;)
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 06:50:20 PM
I see occasional hanging connections when I refresh the wiki, specifically http://arcengames.com/mediawiki/index.php?title=Stars_Beyond_Reach_Beta_Release_Notes which is particularly interesting to me at present. This happened before the migration as well, with about the same frequency, but I don't remember it being a problem during the TLF beta which was the last time I hammered the wiki like this.

I don't always get the hanging connections, but when I do it's pretty consistently on one or two of these resources:
  • http://arcengames.com/mediawiki/load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint%2Cshared%7Cskins.monobook&only=styles&skin=monobook&*
  • http://arcengames.com/mediawiki/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=monobook&*
  • http://arcengames.com/mediawiki/skins/common/images/poweredby_mediawiki_88x31.png

Without looking at the network tab of Chrome's developer tools, it just looks like a very slow page load and might be related to what Chris was seeing. If I hit cmd+shift+R to bypass caching/last-modified (thanks for the hint about caching), I get a pretty snappy load. If I hit cmd+R for a normal reload, I get the hanging connections more often than not. Those look like scripts, CSS, and images, which shouldn't be changing at all between page loads.

Hope this helps you debug this. I'm going to just use cmd+shift+R on the wiki from now on. ;)

Heh, I never have the hanging connections, but I also macroed cmd+shift+R and only use that for reload. :)

I don't know if you noticed, but at the bottom it gave an error about a folder not being writable, which you won't see now, because it is writable. So, if I am correct, that helped immensely and should have fixed a lot of the hanging connections. I did do a normal refresh and it wasn't hanging for me, but that might also just be me.
Title: Re: Site Maintenance
Post by: jerith on May 25, 2015, 07:54:14 PM
I'm still getting the connection hangs, always on poweredby_mediawiki_88x31.png or one of the load.php?blahblah resources. Maybe it has something to do with me and the local cloudflare endpoint being 200ms from the rest of the internet.

If I use HTTPS I seem to get fewer hangs, and only in the load.php?blahblah resources that are fetched over HTTP. I have no idea if that's relevant at all, but I've just noticed that it's nearly 2am and I have to write some messy datastore index migration code tomorrow.

Thanks again for your work on this stuff.
Title: Re: Site Maintenance
Post by: Dominus Arbitrationis on May 25, 2015, 09:32:07 PM
Hm then I'll look into pages like that and see if something is messed up there. Strange that SSL helps, usually it does the exact opposite.

If anyone is lagging right now it is because I'm doing some heavy stuff with the web server, sorry!
Title: Re: Site Maintenance
Post by: x4000 on May 26, 2015, 08:42:08 AM
I'm still getting the connection hangs, always on poweredby_mediawiki_88x31.png or one of the load.php?blahblah resources. Maybe it has something to do with me and the local cloudflare endpoint being 200ms from the rest of the internet.

If I use HTTPS I seem to get fewer hangs, and only in the load.php?blahblah resources that are fetched over HTTP. I have no idea if that's relevant at all, but I've just noticed that it's nearly 2am and I have to write some messy datastore index migration code tomorrow.

Thanks again for your work on this stuff.

I don't know how you're finding out exactly what resources it's hanging on, but that's great information to have.  I am having the exact same issues as you anyhow, though, and I'm pretty near the main eastern US backbone, so my ping times tend to be excellent.
Title: Re: Site Maintenance
Post by: jerith on May 26, 2015, 01:23:12 PM
If you open Chrome's developer tools (cmd+alt+I on OSX, I'm guessing ctrl+alt+I on Windows) and go to the Network tab you'll see all the network connections the browser makes when you reload the page along with how long they took, etc.

Very handy for this sort of thing, even though I try to avoid having to use tools like this. ;)
Title: Re: Site Maintenance
Post by: x4000 on May 26, 2015, 01:32:43 PM
Nice!
Title: Re: Site Maintenance
Post by: Draco18s on May 26, 2015, 04:31:19 PM
Firefox has similar tools.
Right-click -> Inspect Element, Network tab.