Google reported yesterday that their search engine will now include a website’s speed in the list of factors it uses to decide how high to rank the site in its search results.  In this post I consider what this means for web developers and what steps you can take to make your site faster.

Google has made it clear that speed is not the primary factor when determining page rank:

While site speed is a new signal, it doesn’t carry as much weight as the relevance of a page. Currently, fewer than 1% of search queries are affected by the site speed signal in our implementation and the signal for site speed only applies for visitors searching in English on Google.com at this point.

It is, however, a turning point for web site developers.  What Google is really doing is putting a tangible value on the User Experience.  A fast site makes for a more satisfactory experience, and linking to fast sites also makes users happier with Google.  I know how frustrating it is to see what appears to be a promising search result on Google, only to find the site very sluggish once I click through Google’s recommendation.  So, like it or not, you and I will now have to make sure we at least consider our sites’ rendering speeds.

As I had mentioned in a earlier post, I recently changed the taskforest.com website to static HTML from a template-based system.   After reading Google’s blog entry, I installed Firebug on Firefox, as well as two different page performance measuring tools: PageSpeed and YSlow from Yahoo and started testing out the TaskForest web site.  To my dismay, the site scored a paltry 76/100 on PageSpeed and a ‘D’ (67/100) on YSlow.  I’m glad to say that in less than an hour I was able to make the changes that most significantly improved my site’s speed.

There were four major areas in which the two tools recommended changes:

Exploiting Browsers’ Caches

The site was not using the Expires response headers. As this document from Google shows, proper usage of this HTTP header can greatly reduce load on the server as well as clients.The Expires header tells the client how long the server expects the resource to stay unchanged.  For images like logos and screenshots that don’t change very often, this period could (and should) be as high as few months, possibly even the maximum allowed value – one year.  For HTML pages that change often, you could set it to 2-7 days.

The taskforest site which was using Apache 1.41, was not sending the Expires HTTP response header. The reason is that the module that sets this, mod_expires, is not enabled by default.

Combining External CSS

The website was using three CSS style sheets.  One was my own ‘styles.css,’ but the other two were Yahoos YUI style sheets.  For every request for a web page, there were 3 additional requests for style sheets.

Combining External Javascript

Just like style sheets, There were 6 Javascript files that were being served with every page.

Using Compression

The website was not compressing the data it sent to the client.  As this article and the video below from Google shows, data compression can have a major impact on the size of the payload the server delivers to the client, thus making the site faster.

Optimizing the site

So the first order of business was to enable mod-expires and install and activate mod_gzip.  These are the Apache modules that enable the Expires header and Response compression.  I reconfigured Apache with the –enable-module=expires and –activate-module=src/modules/gzip/mod_gzip.c options.  Previously, the relevant lines in my httpd.conf file looked like this:

<VirtualHost 216.139.227.47>
 ServerName www.taskforest.com
 DocumentRoot ".../taskforest/htdocs/website"
</VirtualHost>

After installing the two new modules, my httpd.conf file looked like this:

<VirtualHost 216.139.228.44>
 ServerName www.taskforest.com
 DocumentRoot ".../taskforest/htdocs/website"

 FileEtag None

 # enable expirations
 ExpiresActive On

 # unless overridden elsewhere resources expires
 # after 60 days in the client's cache
 #
 ExpiresDefault "access plus 60 days"

 # The site's feed expires after 1 day in the cache
 ExpiresByType application/xml A86400

 # HTML files expire after 1 day in the cache
 ExpiresByType text/html A86400

 # Turn compressing on
 mod_gzip_on Yes

 # Don't bother compressing tiny files
 mod_gzip_minimum_file_size  1002

 mod_gzip_maximum_file_size  0
 mod_gzip_maximum_inmem_size 60000

 # Compress XML files, text files (including HTML)
 # and directory listings
 mod_gzip_item_include mime "application/xml"
 mod_gzip_item_include mime text/*
 mod_gzip_item_include mime "httpd/unix-directory"

 # Compress files based on their names as well
 mod_gzip_item_include file "\.txt$"
 mod_gzip_item_include file "\.html$"
 mod_gzip_item_include file "\.css$"
 mod_gzip_item_include file "\.js$"

 mod_gzip_dechunk Yes
 mod_gzip_temp_dir "/tmp"
 mod_gzip_keep_workfiles No

</VirtualHost>
<VirtualHost 216.139.228.44>
 ServerName taskforest.com
 RedirectMatch (.*) http://www.taskforest.com$1
</VirtualHost>

For the multiple CSS files, I just copied both of the YUI  style sheets into my styles.css file, preserving the copyright comments.  As for the YUI javascript files, it turns out that none of them were being used!  They were just copied and pasted from the taskforestd webserver web site, which does use the javascript files.  I was able to delete them altogether.  It was rather embarrassing to realize that the site was serving unused files.  But I was glad to have the tools that helped me realize this.

Results of Optimization

The results of the optimization were startling.  The PageTest scores originally looked like this:

PageSpeed Results Before Optimization

Now, after the optimization, the report was much more respectable:

PageSpeed Results After Optimization

This exercise gave me new insight into what is needed to make a website faster.  It’s not just the fastest machine and the most optimized database that matters.  One also has to consider the fundamental characteristics of the Web, and HTTP in particular, where a little effort can bring significant improvement.

Setting up a Sandbox Domain

The first step in trying out WordPress was to set up a new domain just for testing out the WordPress installation.  This way, I wouldn’t affect the taskforest.com domain during my experiments.  I happen to run my own name servers using Daniel Bernstein’s tinydns, so I decided to create a new domain called tf.enoor.com, a subdomain of my defunct company’s domain.  Since I use bluehost.com’s WordPress hosting service, I had to make their name servers responsible for the domain.  All that’s needed is adding the following two lines to /etc/tinydns/root/data:

&tf.enoor.com:74.220.195.31:ns1.bluehost.com:300
&tf.enoor.com:69.89.16.4:ns2.bluehost.com:300

Selecting a Theme

After setting up WordPress, the next step was to select a theme.  I wanted something similar to TaskForest’s current design, so I chose the remarkably customizable Suffusion theme.  After some tinkering I was able to get a site that was quite similar to the original, with one compromise – I could get either the logo or the site’s name in the header, but not both.  Not the way they are on the current site.  It’s quite important that the site’s name appear in the header, because that helps with Search Engine Ranking.  So with a heavy heart I decided to omit the logo.

The Problem with Content

TaskForest ships with its own web server to support REST interface.  As part of the included website is all the documentation for the system. Just like the code, this documentation is under source control, and it’s also used to populate the taskforest.com website.  This way, I can ensure that both the taskforest.com website and a user’s local install have the most up-to-date docs, as long as the user is running the latest version of the software.  What this also means is that I have a few dozen webpages that need to be transfered to WordPress before the new site can go live.  I was already resigned to the fact that the URLs of these pages would be different – the current site has URLs that look like http://www.taskforest.com/about.html, but the default WordPress installation would use URLs that look like http://www.taskforest.com/about/.  It’s not a huge deal, but I prefer my way.

The bigger issue is that when a new version of the software is released, the pages change.  The current build process ensures that the client website and the taskforest.com website stay in sync.  Now if I use WordPress, I don’t want to manually edit the pages using the WP admin site.  I need to install a new plugin that handles inclusion of files.  So, I installed the WP Include plugin.  I’d have to change my build process, but I could get it to work.

Putting It All Together

Okay, so I got the themes and plugins installed, and I’ve got the process worked out.  It was then time to try a proof of concept with a single page.  It worked just as expected, but the site seemed very sluggish.  I made sure I didn’t have anything enabled that I didn’t need.  Still, the site was noticeably slower than the existing site, and that’s with only one non-blank page and zero blog posts.  I thought that maybe the problem was that the bluehost.com shared server was too slow.  I just happened to have an unused server in the same data center that hosts taskforest.com.  The same kind of server as well. It took the better part of the morning, but I installed PHP, mysql and WordPress on that server.  In installed PHP as a static module within Apache, for optimal performance.  Even then, on a pristine machine running nothing else, it was slow.

How Slow is Slow?

Anyone who knows me knows that I’m an Engineer, and as an Engineer I like metrics.  I wanted to know that it wasn’t just my own bias that was penalizing WordPress over my established way of doing things.  So, I looked at the Safari web browser on the Mac.  Safari has a really useful feature called the Web Inspector that, among other things, displays the amount of time it takes for different parts of a page to load.  The numbers were very surprising.  With WordPress, a new page would take 1.75 seconds to load – an eternity on a high-speed broadband connection.  A subsequent request of the same page would take about 700ms. Switching to the TaskForest website’s strategy would take about 800ms for a new page, and 275ms for a cached page.  That’s more than a 200% increase in speed!  The TaskForest website used a preprocessor that allowed for including header files.  It ran under mod_perl.  I decided to write a spider that crawled the website and convert everything to static pages, and serve those pages without mod_perl.  With this enhancement, the results were even better.  I saw a 400% to 800% speed increase, with cached pages taking just 140ms to load.

One of the reasons WordPress’s numbers were so poor is that if articles like this one are still correct, WordPerfect does not cache the content of pages by default.  Resources within the HTML page, like images or javascript files seem to be cached, but every page still does upto 120 database accesses depending on the setup.  Apache is a lot better at caching requests for static HTML pages.  Even the TaskForest webserver used for the REST API, caches data very intelligently using the HTTP headers.

What Does This All Mean?

Given the amount of compromises I would have to make in the design, layout and build process, switching to WordPress would have been very painful.  However, with the performance hit the site would be taking, it seems unlikely that I’ll switch to WP any for the TaskForest website any time soon.  I think I’d rather write a script to generate RSS feeds on demand and automatically submit the feed to feed notification sites like pingomatic. I think this was a great learning experience that will help me with similar decisions in the future.  And I also got to write a cool spider and increase the speed of the TaskForest web site.

Safari's Web Inspector

There are, however, many classes of applications for which text files are the preferred means of storing data.  One of the main reasons is that when data is stored in a relational database, editing it is not a trivial task.  A well-normalized database is not easily updated via an SQL command line.  More often than not, a dedicated, graphical editor is needed to model the complex relationships.

Several years ago, when I wrote TaskForest, one of the initial design requirements was that it be easily configurable with just a shell prompt and one’s favorite text editor. Many of the servers I cared for for schools and non-profits were old boxes which I administered by logging into them via ssh. So when it came to designing job definitions and dependencies, I chose a text file representation. The benefits of text files over a graphical user interface for this include:

Easy Remote Access

All you need is the ability to get to a command line and a text editor on the machine that holds the data files. With the such low client access requirements, virtually any old machine that has internet access and an ssh client can be used to administer the system. I have often worked on my own taskforestd server from a local internet cafe using a Putty.exe downloaded minutes earlier.

Mobile Access

Text files also make work relatively easy using a mobile ssh client like Idokorro Mobile SSH. A dedicated mobile client would be ideal, but short of that, the text file approach assures low bandwidth usage and easy-to-make changes.

Flexibility

The simple, easily parseable format of text files allows us to build richer graphical clients later that would use a graphical interface to specify relationships between jobs.

Source Control

The text based format makes it easy to place the data files under source control. You can also easily diff different versions of the same data file.

Grep

When you have dozens of job group files and hundreds of jobs, you may need to answer questions like: “Are we still running Job J?” This can easily be answered by grepping the files for job J.

Low footprint

When you’re designing an open-source application you may want to minimize the complexity of the system by not forcing dependencies on major subsystems like GUI libraries and Relational Databases.  Of course there is a point at which a such dependencies are inevitable – you have to periodically re-evaluate your decisions and determine whether decisions that were correct, say, a year ago are still correct today.

Choosing A Text Format

In the case of the TaskForest project, the most difficult task by far was choosing which text format to use.  I went through several iterations trying to find one that was simple to read and write, and yet rich enough to model the domain space completely.  What worked for me (and might work for you) was to ask myself how I would represent the data I’m trying to save given just a pencil and paper.  Drawing in a notebook gave me the flexibility to sketch and edit easily, and once I had a good representation, converting that to a text file was a simple task.

This got me thinking about the benefits of using third-party libraries in our own code.  Why should a program depend on so many external modules when it could implement what it needs itself?

The answer, in one word, is “Testing.”  A well-written piece of software isn’t considered complete if it doesn’t have well-documented tests that exercise every part of the program.

There are many common tasks that most programs perform – such as reading configuration food, performing date arithmetic and writing to log files.  Every popular language has third-party libraries that perform these tasks correctly and reliably.  If I were to try to duplicate them, it would almost certainly result in inferior code; the mature libraries have had the benefits of extensive testing as well as enhancements and fixes of bugs reported by users.  Using third-party software allows me to focus on my program and what makes it unique.

Perl can be a little stubborn, sometimes.