GWJ bugs, feature requests, and updates

Yeah, honestly, I'm surprised the place has run as well as it has for so long on just a Linode. Those databases must be getting gigantic by now.

I'm sorry I don't have enough expertise to make suggestions... all I can do is handwave about optimizing queries, which is hardly useful.

I really thought InnoDB would be a great step to take, but clearly had some issues. It tested fine on my dev instance, but under any significant type of load (even a Sunday afternoon), it was choking. Disk I/O was definitely higher on InnoDB... several orders of magnitude higher. It could require more tuning, but I think we just don't have the resources to run it really efficiently (e.g. mostly in memory) due to the sheer size of the content-related tables. It's interesting, because conventional wisdom, particularly in the Drupal community, is to use InnoDB, but I think most sites that are doing it are either a) small, and therefore not drawing upon significant amounts of data or b) enterprise, and having the money and resources to make InnoDB purr.

@trueheart78, I may try that over the next week or so... tuning just a few of the tables that churn a bit more than the others. After watching the slow query log for a bit today, all the slowdown was coming from queries to the comments table, which is our biggest... so I won't start there

Is GWJ hosted in Dublin? Amazon's EC2 and MS's BPOS was knocked offline at the same exact time as GWJ went down for the power outage.

http://hardware.slashdot.org/story/1...

Well, you are running on a virtual machine, and virtual machines typically have fairly slow I/O. CPUs virtualize nicely, and RAM partitions cleanly among guest OSes, but almost all the other hardware that guests see is emulated, rather than virtualized. This requires a hurricane of context switches as guests try to write to their "hardware". The host OS traps the I/O fault, figures out what the guest was trying to do, emulates whatever function it was, and then returns the result to the guest. This is terribly, terribly slow.

One way that hypervisors get around this is with paravirtualization. I believe Linode uses Xen, which pioneered the concept. In an emulated environment, the guest doesn't know it's a guest; when paravirtualized, it knows it's a virtual machine, and runs special I/O drivers that reduce the context switching a ton. So make sure you're running the paravirt driver for Xen; if you aren't, that would explain the bad performance all by itself.

Even with paravirtualization, there's still a fair bit of extra overhead, and you also have to split the drive or drives with the other guests on the box. It's still kinda slow, but it's much faster than regular emulation.

You're probably already running that. If so, I think you should consider moving to real hardware. You wouldn't likely need anything insanely powerful. Even a 7200 RPM SATA drive is going to be much faster than what you have now. What you REALLY want is a 15K RPM SCSI drive, or even an enterprise-level SSD, but those are expensive, especially the SSDs.

You can rent prebuilts from places like The Planet. This is much more expensive than building one yourself and shipping it off to someone's colo, but renting a prebuilt means that the provider needs to fix it if it breaks. And they'll have lots of spare parts on hand, so you should rarely have downtime of more than an an hour or two, should something go south. If you don't have any colo expertise on staff, this is probably the way to go.

Costs range from about $100/mo for a weak machine up to about $350/mo for a fully loaded megamemory SCSI beast. Even a weak machine would be much better than a Linode, at least as far as running a database is concerned. Be careful to check the bandwidth allocation in your package; that could be a problem for GWJ, which I imagine moves a hell of a lot of traffic.

Could we get a [ youtubeHD ] tag option to embed the HD version of a video, so you wouldn't always have to click out of the site to see the better version?

That would make people who don't want to spend the bandwidth on the better version have to click out. Aren't the embedded videos a wee bit small for HD quality, in any case?

Hypatian wrote:

That would make people who don't want to spend the bandwidth on the better version have to click out. Aren't the embedded videos a wee bit small for HD quality, in any case?

But which do we have more of, people with decent broadband, or people with low caps or low bandwidth?

I've just seen it on other forums, and much prefer it to the postage stamp embedded windows I never use.

There's no one-size-fits-all solution. I wouldn't be surprised if there's some userscript out there that can alter the YT embedding though.

MannishBoy wrote:
Hypatian wrote:

That would make people who don't want to spend the bandwidth on the better version have to click out. Aren't the embedded videos a wee bit small for HD quality, in any case?

But which do we have more of, people with decent broadband, or people with low caps or low bandwidth?

I've just seen it on other forums, and much prefer it to the postage stamp embedded windows I never use.

Considering the amount of flak a 2mb gif gets in the post a picture thread, I'm pretty sure some way to easily toggle between the two would be best.

trueheart78 wrote:
MannishBoy wrote:
Hypatian wrote:

That would make people who don't want to spend the bandwidth on the better version have to click out. Aren't the embedded videos a wee bit small for HD quality, in any case?

But which do we have more of, people with decent broadband, or people with low caps or low bandwidth?

I've just seen it on other forums, and much prefer it to the postage stamp embedded windows I never use.

Considering the amount of flak a 2mb gif gets in the post a picture thread, I'm pretty sure some way to easily toggle between the two would be best.

That sound good, too. That would be the best option to have a user preference if it could be done on that level.

I guess my thing with the embedded is that people that want the opposite sized version of the clip would just have click the direct youtube link to get the version they want. So unlike large images, they aren't being forced to download more than the image on the video when they click into the thread.

Maybe I'm in the minority in that I can't stand the little vids and want to see a bigger version on my large monitors. If that's the case, I'll just deal with it. But if more people are like me and always click off the site to watch the vid, maybe it can be a tag level option when you post the vid if it's worth seeing in HD?

Not a big deal either way, but I do prefer how youtube/youtubehd tags work on the site I've seen it on.

I don't know if it's just personal preference, but I find most videos that are in those little boxes are 'good enough' for most things. If I'm pushing the resolution for something detailed/pretty enough to need it, it wants it's own window anyway.

In fact, one of the things I like about firefox is that you can tear out a tab and position the video so you can watch it while doing something else - what I'd really like is to pull out a flash player just for that video in it's own resizeable window, detaching the video from the webpage.

I'm sure there's something here to help a bit

Scratched wrote:

I don't know if it's just personal preference, but I find most videos that are in those little boxes are 'good enough' for most things. If I'm pushing the resolution for something detailed/pretty enough to need it, it wants it's own window anyway.

In fact, one of the things I like about firefox is that you can tear out a tab and position the video so you can watch it while doing something else - what I'd really like is to pull out a flash player just for that video in it's own resizeable window, detaching the video from the webpage.

Oh you can move the tab out in Chrome and IE as well, but the little embeds here don't even allow you to full screen the vid and go HD without clicking through to youtube. And even if you had the bigger embeds, you could still do it your way if you want, but you'd have a better image in the embed static image.

*shrug*

Just thought I'd throw it out there as a "nice to have" type of thing.

doogiemac wrote:
Malor wrote:

Yeah, it's gotten slow again. Hmm. I wonder if your new tables are just using more RAM than the old ones? The way the site speeds up and slows down feels kind of like it's hitting swap.

That's what's so strange... not swapping at all. For now, I've gone back to MyISAM (the change I made last night was to switch to InnoDB), and we seem to be snappy again. Trouble is that MyISAM is less stable and prone to crashing. I'm guessing there may be a deeper issue, e.g. physical hardware on the server. We're on a VPS, so I can't directly see what's happening there. The other issue is that InnoDB is inherently slower, particularly when you get to a scale of data we have in a couple of our tables. Goodjers are chatty ;)

MyISAM is much quicker, but it lacks some of the foreign key support and other types of indexes. When our system started doing this it turned out to be only a couple of queries that were specifically running slow. We added a few indexes which increased the disk usage, but overall sped up the performance greatly.

Also, I can agree with Malor on the virtualization. We run a complete virtual environment, however, our database is hosted on a dedicated machine while our web servers work off the virutalized system. Our rule of thumb here is high disk IO gets you a full host, while ram/cpu intense code goes virtual.

Good luck debugging the site!

Speaking of clicking through to youtube to watch the video, does anyone ever have the video keep playing on the GWJ page while the youtube page loads and starts playing? It doesn't always do it for me, but when it does, it's annoying. Mostly when lurking in the post a video thread. Using Firefox, and it's done it on Win 7(64) & XP.

mrtomaytohead wrote:

Speaking of clicking through to youtube to watch the video, does anyone ever have the video keep playing on the GWJ page while the youtube page loads and starts playing? It doesn't always do it for me, but when it does, it's annoying. Mostly when lurking in the post a video thread. Using Firefox, and it's done it on Win 7(64) & XP.

I've seen it at times.

Hmm. Compared to the messages you get when you receive a forum PM, the ones you get from the "Contact" page to send an email are awfully fishy looking. On the one hand, I understand the idea: send an email, giving your email address, and not revealing the email address of the recipient. But on the other, I almost deep-sixed the message I got because I thought it might be some sort of really strange phishing attempt.

It might be better to remove that feature and just direct people to send PMs on the forum, which are significantly easier to identify and to check that they're coming from a legit source.

Hypatian wrote:

Hmm. Compared to the messages you get when you receive a forum PM, the ones you get from the "Contact" page to send an email are awfully fishy looking. On the one hand, I understand the idea: send an email, giving your email address, and not revealing the email address of the recipient. But on the other, I almost deep-sixed the message I got because I thought it might be some sort of really strange phishing attempt.

It might be better to remove that feature and just direct people to send PMs on the forum, which are significantly easier to identify and to check that they're coming from a legit source.

You should be able to turn off the personal contact form by editing your account settings. If you don't see the option there, let me know.

Can we block any post or thread title with (ID:NNN) in it? It seems to be what most spam post/threads use now-a-days.

doogiemac wrote:
Hypatian wrote:

Hmm. Compared to the messages you get when you receive a forum PM, the ones you get from the "Contact" page to send an email are awfully fishy looking. On the one hand, I understand the idea: send an email, giving your email address, and not revealing the email address of the recipient. But on the other, I almost deep-sixed the message I got because I thought it might be some sort of really strange phishing attempt.

It might be better to remove that feature and just direct people to send PMs on the forum, which are significantly easier to identify and to check that they're coming from a legit source.

You should be able to turn off the personal contact form by editing your account settings. If you don't see the option there, let me know.

Well, I don't *mind* getting contacted personally. I'm just noting that I'd never even noticed the option before, and my first reaction upon receiving a message from it was to think it was somebody trying to spearphish me, because the message format was so unusual. I think the site could avoid some confusion for people in the future by simply not having that option, since most people use the PM system. Having two different ways to communicate with people privately seems a bit silly.

I've have a small suggestion that's related to image esthetics . Due to the fact HD and higher resolution screens are common some screenshots can hog a lot of thread space. A lot of people don't use full screen to browse the internet and even if they did the screenshots may take the entire screen.

The simplest solution is using the img=320X240 or something similar(people can start doing that now ). It won't reduce the image storage size (big download) but it's presentation size would be fixed. You can either incorporate it to image button or do some post processing (can annoy posters) . Wrapping the img tag with a url tag to it would let people click it and enlarge.

You can probably add a CSS tag and give it using mouse over hover or something. Using CSS would probably be much better. People can then select the presentation they want. People with a bandwidth cap can set it to "replace image with links" which would prevent images from hogging bandwidth unless they are interested in viewing them. You can also set a different setting for mobile browsers and mention the image size in the links .

Speaking as someone stuck with horribly low bandwidth, I'm happy with the low res videos. If it is something that really needs HD, fair enough, but for most things, it would just take too long to stream to be worth waiting for.

Niseg wrote:

I've have a small suggestion that's related to image esthetics . Due to the fact HD and higher resolution screens are common some screenshots can hog a lot of thread space. A lot of people don't use full screen to browse the internet and even if they did the screenshots may take the entire screen.

The simplest solution is using the img=320X240 or something similar(people can start doing that now ). It won't reduce the image storage size (big download) but it's presentation size would be fixed. You can either incorporate it to image button or do some post processing (can annoy posters) . Wrapping the img tag with a url tag to it would let people click it and enlarge.

You can probably add a CSS tag and give it using mouse over hover or something. Using CSS would probably be much better. People can then select the presentation they want. People with a bandwidth cap can set it to "replace image with links" which would prevent images from hogging bandwidth unless they are interested in viewing them. You can also set a different setting for mobile browsers and mention the image size in the links .

It seems like sometimes the first time I view images in a thread here it might be wider than the normal thread width. But then if I refresh the page, or come back to the topic, the image has been resized to fit the width of the forum. So generally it looks ok.

Does it not do that for you?

Speaking of image size, the mobile version seems more affected by extra wide images. Seems like in that version it would be nice to force a max width in a common smart phone resolution (landscape) with a click to enlarge.

But then if I refresh the page, or come back to the topic, the image has been resized to fit the width of the forum. So generally it looks ok.

That's a browser feature; because the image is already in cache, it can scale the image properly to begin with.

If you encode the actual size of the image in the tag, as in {img=640x480}myimageurl{/img}, then the browser will get the sizing correct on the first load, instead of needing a refresh.

Malor wrote:
But then if I refresh the page, or come back to the topic, the image has been resized to fit the width of the forum. So generally it looks ok.

That's a browser feature; because the image is already in cache, it can scale the image properly to begin with.

If you encode the actual size of the image in the tag, as in {img=640x480}myimageurl{/img}, then the browser will get the sizing correct on the first load, instead of needing a refresh.

Of course, it would also be possible to use a CSS style to do this instead of javascript that can't activate until the image finishes loading...

I'm not sure how it actually translates the img tag to HTML, but if you include the size in the tag, it'll always get it right without any apparent redraw or refresh. If the site is using Javascript to send the sizing information, it's not visible to the untrained eye.

Malor wrote:

I'm not sure how it actually translates the img tag to HTML, but if you include the size in the tag, it'll always get it right without any apparent redraw or refresh. If the site is using Javascript to send the sizing information, it's not visible to the untrained eye.

True, but only if *all* images on the page have sizes, or load quickly. Also, I'm always a fan of doing less in javascript.

Let's take a look at:

[img]http://www.arcade-history.com/images...

The code translates it to this:

<img src="http://www.arcade-history.com/images..." alt class="bb-image">

if given just the URL, or if you include the size:

[img=410x256]http://www.arcade-history.com/images...

it becomes this:

<img width="410" height="256" alt src="http://www.arcade-history.com/images..." class="bb-image">

Then there's a function that gets called when the page is "ready" (i.e. fully loaded). If there are less than 100 unique images on the page, then it computes the size of posts, and for every image that does not have "gamerswithjobs.com" in the URL and which is wider than the size of posts, it computes a new width and height for a width that fits within the post, and re-sets the width and height.

The width and height on the img tag don't actually change this process at all--if you give it a huge size, it will still have to resize it--although the page may be "ready" earlier, if there are no other large images on the page. If you do give a width and height, it's possible to give a different aspect ratio than the actual image. If the page is resized after it's loaded and the resize function runs, images will not be resized again, even if there's more or less room than there used to be.

The CSS solution would be to add this rule to the stylesheet:

img.bb-image { max-width: 100% }

This would resize images even while they are loading, and would also resize them if the page size is changed after loading. It does not, however, maintain aspect ratio *if* you have specified a width and height for the image to begin with. (i.e. if you give a width and height to the image max-width would respect the height but change the width to fit on the page.) A solution would be to assume that aspect ratio should always be that of the original image. This style would do that:

img.bb-image { max-width: 100%; height: auto }

With that short style rule, the width given in the img tag will be respected (if it is smaller than the original image or the width of the area it's going into), the height will be ignored to preserve the original image aspect ratio, and if the image is too big it will be shrunk to fit into the available space.

This technique doesn't work directly in IE6 or less, although it's possible to include a short javascript file that will make it work correctly on something that old, or to make the current solution run only on IE versions before 7.

Hah, well, shows you how little I know about what the GWJ server is actually doing. I had no idea that a redraw was even happening. I never see it flicker.

Oh, and:

If there are less than 100 unique images on the page, then it computes the size of posts, and for every image that does not have "gamerswithjobs.com" in the URL and which is wider than the size of posts, it computes a new width and height for a width that fits within the post, and re-sets the width and height.

I'm not sure that's working as intended. It's fairly unusual for a large image to be sized correctly on this site, unless it has the height and width in the image tag.

I think for the really big ones, there may be some timeout that's occurring that prevents the script from firing. Certainly, since the very large images take a really long time to load, it's a long time before the ready event fires. I believe the more proper thing to do for the javascript solution would be to set up a load callback on the document which checks if each image is loaded, if so resizes it, and if not adds a load callback to the image to resize it. *ponder* Not entirely sure, because the image size should be available before the image is fully loaded.

But the CSS solution should be the better solution for what it does now.

The absolute best approach would probably be to do the CSS thing, only even more-so: set a max-width *and* a max-height, so that tall skinny images also get shrunk. Then use a javascript handler attached to all of the embedded images in posts that does a lightbox when you click on the image.

Edit: Aha! I think I know the problem. I think that for really really large images, ready fires on the document before all the images are loaded. So when the script runs to resize images, it can't find their sizes and doesn't do anything. So that script shouldn't be doing the image resize from inside Drupal.gwjmodAutoAttach, which is registered on ready, it should be something else that runs on load. Then it will only happen once all the images are loaded, but it will happen to all of the images once they are loaded. Alternatively, in the ready callback a load callback could be added to each image that resizes only that image when that image is fully loaded.

I can provide a patch to gwjmod.js to do that should it be desired. Then it'll still use javascript to do it with all of the problems I mentioned, but it'll at least work consistently.

Hypatian wrote:

Edit: Aha! I think I know the problem. I think that for really really large images, ready fires on the document before all the images are loaded. So when the script runs to resize images, it can't find their sizes and doesn't do anything. So that script shouldn't be doing the image resize from inside Drupal.gwjmodAutoAttach, which is registered on ready, it should be something else that runs on load. Then it will only happen once all the images are loaded, but it will happen to all of the images once they are loaded. Alternatively, in the ready callback a load callback could be added to each image that resizes only that image when that image is fully loaded.

I can provide a patch to gwjmod.js to do that should it be desired. Then it'll still use javascript to do it with all of the problems I mentioned, but it'll at least work consistently.

Nice. Hypatian, are you a part of the GWJ dev group? If not, and you're interested in being in it, let me know and I'll get you added to the the right group and in Github.