Google reported yesterday that their search engine will now include a website’s speed in the list of factors it uses to decide how high to rank the site in its search results.  In this post I consider what this means for web developers and what steps you can take to make your site faster.

Google has made it clear that speed is not the primary factor when determining page rank:

While site speed is a new signal, it doesn’t carry as much weight as the relevance of a page. Currently, fewer than 1% of search queries are affected by the site speed signal in our implementation and the signal for site speed only applies for visitors searching in English on Google.com at this point.

It is, however, a turning point for web site developers.  What Google is really doing is putting a tangible value on the User Experience.  A fast site makes for a more satisfactory experience, and linking to fast sites also makes users happier with Google.  I know how frustrating it is to see what appears to be a promising search result on Google, only to find the site very sluggish once I click through Google’s recommendation.  So, like it or not, you and I will now have to make sure we at least consider our sites’ rendering speeds.

As I had mentioned in a earlier post, I recently changed the taskforest.com website to static HTML from a template-based system.   After reading Google’s blog entry, I installed Firebug on Firefox, as well as two different page performance measuring tools: PageSpeed and YSlow from Yahoo and started testing out the TaskForest web site.  To my dismay, the site scored a paltry 76/100 on PageSpeed and a ‘D’ (67/100) on YSlow.  I’m glad to say that in less than an hour I was able to make the changes that most significantly improved my site’s speed.

There were four major areas in which the two tools recommended changes:

Exploiting Browsers’ Caches

The site was not using the Expires response headers. As this document from Google shows, proper usage of this HTTP header can greatly reduce load on the server as well as clients.The Expires header tells the client how long the server expects the resource to stay unchanged.  For images like logos and screenshots that don’t change very often, this period could (and should) be as high as few months, possibly even the maximum allowed value – one year.  For HTML pages that change often, you could set it to 2-7 days.

The taskforest site which was using Apache 1.41, was not sending the Expires HTTP response header. The reason is that the module that sets this, mod_expires, is not enabled by default.

Combining External CSS

The website was using three CSS style sheets.  One was my own ‘styles.css,’ but the other two were Yahoos YUI style sheets.  For every request for a web page, there were 3 additional requests for style sheets.

Combining External Javascript

Just like style sheets, There were 6 Javascript files that were being served with every page.

Using Compression

The website was not compressing the data it sent to the client.  As this article and the video below from Google shows, data compression can have a major impact on the size of the payload the server delivers to the client, thus making the site faster.

Optimizing the site

So the first order of business was to enable mod-expires and install and activate mod_gzip.  These are the Apache modules that enable the Expires header and Response compression.  I reconfigured Apache with the –enable-module=expires and –activate-module=src/modules/gzip/mod_gzip.c options.  Previously, the relevant lines in my httpd.conf file looked like this:

<VirtualHost 216.139.227.47>
 ServerName www.taskforest.com
 DocumentRoot ".../taskforest/htdocs/website"
</VirtualHost>

After installing the two new modules, my httpd.conf file looked like this:

<VirtualHost 216.139.228.44>
 ServerName www.taskforest.com
 DocumentRoot ".../taskforest/htdocs/website"

 FileEtag None

 # enable expirations
 ExpiresActive On

 # unless overridden elsewhere resources expires
 # after 60 days in the client's cache
 #
 ExpiresDefault "access plus 60 days"

 # The site's feed expires after 1 day in the cache
 ExpiresByType application/xml A86400

 # HTML files expire after 1 day in the cache
 ExpiresByType text/html A86400

 # Turn compressing on
 mod_gzip_on Yes

 # Don't bother compressing tiny files
 mod_gzip_minimum_file_size  1002

 mod_gzip_maximum_file_size  0
 mod_gzip_maximum_inmem_size 60000

 # Compress XML files, text files (including HTML)
 # and directory listings
 mod_gzip_item_include mime "application/xml"
 mod_gzip_item_include mime text/*
 mod_gzip_item_include mime "httpd/unix-directory"

 # Compress files based on their names as well
 mod_gzip_item_include file "\.txt$"
 mod_gzip_item_include file "\.html$"
 mod_gzip_item_include file "\.css$"
 mod_gzip_item_include file "\.js$"

 mod_gzip_dechunk Yes
 mod_gzip_temp_dir "/tmp"
 mod_gzip_keep_workfiles No

</VirtualHost>
<VirtualHost 216.139.228.44>
 ServerName taskforest.com
 RedirectMatch (.*) http://www.taskforest.com$1
</VirtualHost>

For the multiple CSS files, I just copied both of the YUI  style sheets into my styles.css file, preserving the copyright comments.  As for the YUI javascript files, it turns out that none of them were being used!  They were just copied and pasted from the taskforestd webserver web site, which does use the javascript files.  I was able to delete them altogether.  It was rather embarrassing to realize that the site was serving unused files.  But I was glad to have the tools that helped me realize this.

Results of Optimization

The results of the optimization were startling.  The PageTest scores originally looked like this:

PageSpeed Results Before Optimization

Now, after the optimization, the report was much more respectable:

PageSpeed Results After Optimization

This exercise gave me new insight into what is needed to make a website faster.  It’s not just the fastest machine and the most optimized database that matters.  One also has to consider the fundamental characteristics of the Web, and HTTP in particular, where a little effort can bring significant improvement.

I am certain Google tested their application thoroughly.  They’ve been known to do extensive usability tests for the seemingly tiniest of changes to their web site.  But even the most well-implemented tests are incomplete if they’re not performed on a statistically representative sample of the audience.

But today, the BBC reported that Google has admitted that they only tested Buzz internally, and bypassed their regular rigorous testing procedures — possibly in an attempt to get it out the door as soon as possible. I’ll let the pundits decide if it did more harm than good to the firm, but it’s a warning to other software developers: skipping testing can lead to embarrassing failures.

Google Buzz

Almost immediately after Google started rolling out Buzz to its Gmail clients various blogs reported that there was an egregious flaw in the system that exposed people’s private contacts to the rest of the world.  When a Gmail user logged in, they were presented with a dialog box that asked them if they wanted an introduction to Buzz or not.  If the user declined, it didn’t mean that Buzz was disabled – it only meant that they didn’t want the intro.  What Google did behind the scenes was that they scanned the user’s emails and collected the emails of the people with whom the user corresponded most often.  Then, it automatically modified the user’s Gmail account to ‘follow’ those people in Buzz.

But here’s the thing that infuriated most people – it made that list of the people that user’s following public: Anyone who looked at the user’s public Google profile could see who that person was following.  In other words, anyone looking at that user’s public profile could see who that user emailed most often.

Now pretend you’re a reporter who’s communicating with dissidents in Myanmar or Iran.  Would you want your heretofore anonymous sources made public?  In one reported case, a woman’s most frequent contacts were her boyfriend and her mother.  Her third-most-frequent contact (with a lot fewer emails) was her abusive ex-husband, from whom she had kept her email address hidden.  But what Google did when they released Buzz is automatically have her follow her ex, and share her shared Google Reader messages, including her location and workplace with her ex.

Another blogger reported that her Android phone uploaded to her public profile a photo that she had taken, but not uploaded. She thought her photo was private, but Google’s assumption was that if the photo was on her phone, it would be okay to upload it to her public profile.

The third problem that surfaced in the two days after Buzz was released was that in some cases users thought that they were referencing a person’s public email address in a buzz (think of it as a retweet or a reply) but in reality were referencing a private email address.  When they selected a person by their name, it wasn’t clear which email address was linked to that name.  For example, if John Smith’s public email address is john_smith@example.com, and his personal address is js_123@example.com, and the public address is associated with a Google profile, a Buzz user may inadvertently select the private email address in a buzz, thus exposing John’s private address to everyone he follows.

The iPad

In the minutes after Apple’s Steve Jobs announced the new mobile appliance’s name, people started making very funny, original and sometimes vulgar jokes about the name iPad.  I could post them here, but the jokes are old by now.

Salacious Texts On Samsung Virgin Mobile Phones

According to this article, a 49 year old man from Winnipeg can no longer have a relationship with the woman he loved because she accused him of cheating on him.  Her proof?  Sexy text messages on his cell phones.  He denied the affair, but couldn’t explain how they got there.  Only later did he learn that the messages were pre-loaded by Virgin Mobile.

What Do These Cases Have in Common?

As an outsider it appears to me that in each of the three cases the problem was that the population for whom the system was designed wasn’t necessarily the only population actually using the system.  Let’s look at the Google Buzz case.  If we claim that Google is neither evil nor stupid, what went wrong?  I think there were several assumptions that the Google developers and designers made that were incorrect.

Buzz seemed to have be designed for people who collaborate together in tightly knit groups.  The assumption is that people who want to share information with each other communicate a lot with each other.  In real life that isn’t always true, and neither is the converse.  In many cases the people that I want to share the most with are people with whom I have so many channels of communication (including the dinner table) that any single channel like email contains a fraction of the actual communication taking place.  Similarly, as in the case of the lady who was hiding her online identity from her ex, the people we communicate with aren’t always the people we want to share information with.  If Google’s designers approached their product with that bias, they could have made (and did make) a technically excellent product that worked as they intended, and whose test cases passed, but which did exactly the wrong thing.

I think that that’s essentially the core problem with the issue of the publication of private photos and email addresses.  The joke in the blogs was that Buzz was designed for an people like engineers at Google, not for people who go to parties and take pictures that they would regret the next morning (and I know they’re not mutually exclusive groups).  Unlike in the business world and the networks of sites like LinkedIn, people sometimes guard their email addresses jealously and do not want them published without their authorization.

I am certain Google tested their application thoroughly.  They’ve been known to do extensive usability tests for the seemingly tiniest of changes to their web site.  But even the most well-implemented tests are incomplete if they’re not performed on a statistically representative sample of the audience.

The same can be said for Apple and Virgin Mobile.  On of the most insightful questions asked after the iPad announcement was, “Did Apple not have any women on the product naming committee?”  Did Virgin Mobile think that their only consumers would be teenagers who would find the default text messages cute and funny and have the technical expertise to realize that these messages were pre-loaded?  While Apple seemed to have failed to consider about half of the human race while naming their product, Virgin failed to consider the less savvy part of the population who are their customers.

What I don’t know is this: Let’s assume I’m wrong, and that in all three cases a statistically-representative portion of the population was tested or considered.  If that’s true, then could these three cases be considered edge conditions?  While a problem could exist in theory could the actual problems be so few that they’re statistically insignificant?  If you’re the person whose lost a relationship or now have to worry about being stalked by an ex it doesn’t really matter.  It’s affected your life in a significant way.

What Can Software Developers Learn From This?

I don’t think anyone or any company (not even Google) can make the perfect application that pleases everyone.  But I think that there are a few rules of thumb that we should follow, to prevent embarrassing and distressing problems.

First: Applications like these should not take action on behalf of the user.  The system should assume that the user always knows what’s best for them.  The most the program should do is make suggestions that the user can then decide to obey or ignore.  This is exactly what Google did (among other things) to change Buzz after the huge public outcry.

Second: Features, no matter how well-intentioned, no should never be opt-out.  They should always be opt-in.  Buzz was essentially an opt-out system.  You could be vulnerable by not doing anything. It isn’t that way now, but it was for those two days when Google’s reputation took a massive hit.

Third: Designers and service providers need to pay careful attention to who their users actually are.  They’re not necessarily men, not necessarily tech-savvy teens or professionals looking to increase their network.  They could include people very different from us.

As developers and designers we need to realize that the products we make will be used in manners that we never imagined, by people who we don’t understand.  If we give our customers the ability to control how our products and services are used, we’ll be helping them and thus helping ourselves.