GWJ bugs, feature requests, and updates

Jeff-66 wrote:
Malor wrote:

Wow, the server is just falling into a ditch today. The site's barely usable at 10:30 Eastern.

To be honest, I've had issues with GWJ for months. I often have crazy long times 'connecting'. This is not a complaint, just an FYI ... I'm not looking the gift horse in the mouth.

It's weird, sometimes, a page will load in one second, then the next page takes 30 to 45 seconds to connect, then the next might take 5 seconds, then 30, then instantly, then 60 seconds. For me, this has been happening for a long time. I've tried different browsers, and different PC's in the house.

Partly the same, though it's only been today it's taken this long. Instant load, then a lengthy one. Average out at maybe 7-8 seconds. On some days it'll remain fast, but in recent months, it's been more slow than anything else.

Edit: *Right now* it's great.

The server seems underprovisioned. They're really about due to move to real hardware from a VM, but so far have seemed resistant to doing so, perhaps because that's a fair bit more money. (probably about $125-$150/mo, although if they can provide a machine and can admister it almost 100% remotely, I know a spot taking colocated servers for cheap.)

I can be added to the list of long loads. This morning I have been getting 30sec.-1min loads. I have noticed that over the last few months loads have been getting longer and longer. I love the site and a willing to help out with money if it would improve overall performance.

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

I've had long loading times for a while now. And more recently, sometimes pages will just fail to load.

juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

Right, the number disappeared. Something break it?

MannishBoy wrote:
juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

Right, the number disappeared. Something break it?

Same

MannishBoy wrote:
juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

Right, the number disappeared. Something break it?

It may have been disabled for system performance reasons, who knows!

mrtomaytohead wrote:

I've had long loading times for a while now. And more recently, sometimes pages will just fail to load.

Same here.

Demyx is also reporting she's getting 403 forbidden errors some how when trying to hit any part of the site. Does anyone know what might be causing this? Wasn't happening before they took the database down for optimizations...

trueheart78 wrote:
MannishBoy wrote:
juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

Right, the number disappeared. Something break it?

It may have been disabled for system performance reasons, who knows!

Well, it makes me click an extra time when I come into the site to load that page, so I'm not sure if it's a win or not

Yeah that was my favorite part of the favorites!

I know this has been asked before, and maybe I missed the answer, but can we please, please, please, PLEASE get Tapatalk support? I would love you forever long time.

mrtomaytohead wrote:

Demyx is also reporting she's getting 403 forbidden errors some how when trying to hit any part of the site. Does anyone know what might be causing this? Wasn't happening before they took the database down for optimizations...

Can you ask him to try again. I may have been too aggressive in blocking some IPs that looked abusive.

juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

For now, I've disabled that to see if we can eek out some additional performance.

Just some additional notes on performance of the site: First, I've been overwhelmingly busy at work and haven't been able to put the time necessary into maintaining the site at a system level. I'm hoping this will improve soon, but who knows... Second, I've been plugging away at a new build of the site based on Drupal 7 (if you want to help, let me know). This would provide several performance improvements as we could more effectively use InnoDB tables for MySQL, and I could switch to a more performant MySQL-compatible distro, like MariaDB. Switching to D7 also allows us to readily use technologies like Varnish for front-end caching, which takes the load off for any assets like images, css, js, etc. as well as provides highly-performant caching for anonymous page views (unregistered users).

Most importantly, though, the move to D7 puts us in a spot where we can move to solid managed hosting (no one really supports managed hosting for D5 sites). I can't promise anything yet, but the company I work for provides a great managed cloud hosting service that I may be able to swing a discount on, and there are some other great services out there as well. I'm hesitant to go the dedicated hardware route. After years of being reliant on pretty much just me, I don't want Certis to be stuck in a similar situation going forward. A managed service would provide him far great flexibility in dealing with hosting, support, billing, etc. and removes the "what do we do if Doogiemac is hit by a bus?" factor we have right now.

doogiemac wrote:
juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

For now, I've disabled that to see if we can eek out some additional performance.

Understood. Just hope it comes back once things settle. That's one of the most useful things for me that's been added in recent memory. Especially on mobile, where it saves even more time if I just want to get in and get out of the site in a hurry while I'm killing a couple of minutes.

doogiemac wrote:

Can you ask him to try again. I may have been too aggressive in blocking some IPs that looked abusive.

Will do. She was having trouble accessing it from work so it won't be until Monday. I'll tell her to quit hitting F5 too

Most importantly, though, the move to D7 puts us in a spot where we can move to solid managed hosting (no one really supports managed hosting for D5 sites). I can't promise anything yet, but the company I work for provides a great managed cloud hosting service that I may be able to swing a discount on, and there are some other great services out there as well. I'm hesitant to go the dedicated hardware route. After years of being reliant on pretty much just me, I don't want Certis to be stuck in a similar situation going forward. A managed service would provide him far great flexibility in dealing with hosting, support, billing, etc. and removes the "what do we do if Doogiemac is hit by a bus?" factor we have right now.

Sounds like a good plan!

shoptroll wrote:

Will do. She was having trouble accessing it from work so it won't be until Monday. I'll tell her to quit hitting F5 too

I will try to be less abusive!

juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

Ditto, new as of today or yesterday, I think.

Mixolyde wrote:
juv3nal wrote:

"my favorite topics" thingy in the top left I would swear used to show an unread thread count, but doesn't seem to be doing so right now...

Ditto, new as of today or yesterday, I think.

Read down. Doogiematic said it was turned off to try to improve site performance.

Haha, acquia. We use acquia and I work on Drupal 7 full time for work now. Let me know of you need help with host management / site building, etc. I'm on the github project but I don't really want to jump into Drupal 5 world.

Demyx wrote:
shoptroll wrote:

Will do. She was having trouble accessing it from work so it won't be until Monday. I'll tell her to quit hitting F5 too ;)

I will try to be less abusive!

It may not have been you, Demyx. Sometimes we get heavy traffic from a single IP, but it'll be from an ISP (for example). Apologies for locking you out!

----

A couple of days ago, I re-enabled the MySQL slow query log to try and get some idea for what's slow. I spent some time this morning reviewing the slow queries and found most of the slow queries were happening when any node with comments on it, e.g. an article or a forum thread. I believe I've figured out some additional indexes we can put on the comments table (which is now over 2 million rows!), and I validated these by seeing what D7 is doing in terms of keys.

Later this evening, I'll try implementing them and we'll see if that helps. I don't see any reason they'd hurt performance, so it's worth a short. It'll require some downtime, as it takes a while to build indexes that large, so I'll wait until tonight.

PandaEskimo, there's a D7 repo in the Gamers With Jobs project on Github. Feel free to check it out. If you've never used drush make before, the structure may be a little odd. I intend on putting in some documentation when I return to working on it (hopefully late this week / early next).

----

If these fixes work, I'll reinstate the counters on the favorites / inbox for you all

doogiemac wrote:

It may not have been you, Demyx. Sometimes we get heavy traffic from a single IP, but it'll be from an ISP (for example). Apologies for locking you out!

Not a problem Hopefully that's what it was. I'll find out Monday!

Things have been MUCH better for the last day or two. Dunno if the database has been rekeyed yet, but the place is pretty snappy ATM.

So, out of curiosity how big is the GWJ database now?

Dr.Ghastly wrote:

So, out of curiosity how big is the GWJ database now?

Big. Some general stats to give you an idea:

* Over 110k nodes. This means forum posts and articles.
* Over 2.2m comments. Yes, million. This is comments on articles, as well as any post in a forum thread that isn't the first post.

Users are a bit harder to measure, and I hesitate to give that without Certis' feedback anyways. Nodes and comments are easily scrapable, but user data is a little more sensitive.

I work with a lot of Drupal sites on a daily basis, and very few have come close to the sheer amount of content that we have here at GWJ.

doogiemac wrote:

Over 2.2m comments.

Sorry about that.

Ooops, another one.

doogiemac wrote:
Dr.Ghastly wrote:

So, out of curiosity how big is the GWJ database now?

Big. Some general stats to give you an idea:

* Over 110k nodes. This means forum posts and articles.
* Over 2.2m comments. Yes, million. This is comments on articles, as well as any post in a forum thread that isn't the first post.

Users are a bit harder to measure, and I hesitate to give that without Certis' feedback anyways. Nodes and comments are easily scrapable, but user data is a little more sensitive.

I work with a lot of Drupal sites on a daily basis, and very few have come close to the sheer amount of content that we have here at GWJ.

I work in databases that hit > a billion rows on some tables I was just curious if the DB was, say, 50GB?

A heads up re: performance. Last night I added some I indexes to the comments table that should help address some of the slow queries I was seeing in our logs. Hopefully you're all seeing some better performance today.

Sometime this week I'll also run some general application maintenance and get some of the underlying OS stuff updated and tweaked. For now, I'd like to isolate the db changes and see how measurable the improvement is

Well, not counting markup, to get a database of GWJ comments of 50 GB would require 50,000,000,000 characters. At an average of 5.1 characters per English word (estimate from Wolfram Alpha), GWJ would have to contain roughly 9,803,921,569 words. The longest English-language novel, according to Wikipedia, is Clarissa, or, the History of a Young Lady at 984,870 words. GWJ would have to be be slightly less than 1,000 copies of that novel in length. Stacked end-to-end, that would be a stack of books 766 feet tall, or slightly more than one quarter of the Burj Khalifa, the tallest building in the world.

NSMike wrote:

Well, not counting markup, to get a database of GWJ comments of 50 GB would require 50,000,000,000 characters. At an average of 5.1 characters per English word (estimate from Wolfram Alpha), GWJ would have to contain roughly 9,803,921,569 words. The longest English-language novel, according to Wikipedia, is Clarissa, or, the History of a Young Lady at 984,870 words. GWJ would have to be be slightly less than 1,000 copies of that novel in length. Stacked end-to-end, that would be a stack of books 766 feet tall, or slightly more than one quarter of the Burj Khalifa, the tallest building in the world.

Don't discount the size of indexes, be they PK or clustered indexes for improving query performance. I have an index on a table that is 5 GB in size

doogiemac

Just a question for my own curiosity. Do animated .gif pictures posted in topics have any negative impacts on the database as a whole?