Setting up a Sandbox Domain

The first step in trying out WordPress was to set up a new domain just for testing out the WordPress installation.  This way, I wouldn’t affect the taskforest.com domain during my experiments.  I happen to run my own name servers using Daniel Bernstein’s tinydns, so I decided to create a new domain called tf.enoor.com, a subdomain of my defunct company’s domain.  Since I use bluehost.com’s WordPress hosting service, I had to make their name servers responsible for the domain.  All that’s needed is adding the following two lines to /etc/tinydns/root/data:

&tf.enoor.com:74.220.195.31:ns1.bluehost.com:300
&tf.enoor.com:69.89.16.4:ns2.bluehost.com:300

Selecting a Theme

After setting up WordPress, the next step was to select a theme.  I wanted something similar to TaskForest’s current design, so I chose the remarkably customizable Suffusion theme.  After some tinkering I was able to get a site that was quite similar to the original, with one compromise – I could get either the logo or the site’s name in the header, but not both.  Not the way they are on the current site.  It’s quite important that the site’s name appear in the header, because that helps with Search Engine Ranking.  So with a heavy heart I decided to omit the logo.

The Problem with Content

TaskForest ships with its own web server to support REST interface.  As part of the included website is all the documentation for the system. Just like the code, this documentation is under source control, and it’s also used to populate the taskforest.com website.  This way, I can ensure that both the taskforest.com website and a user’s local install have the most up-to-date docs, as long as the user is running the latest version of the software.  What this also means is that I have a few dozen webpages that need to be transfered to WordPress before the new site can go live.  I was already resigned to the fact that the URLs of these pages would be different – the current site has URLs that look like http://www.taskforest.com/about.html, but the default WordPress installation would use URLs that look like http://www.taskforest.com/about/.  It’s not a huge deal, but I prefer my way.

The bigger issue is that when a new version of the software is released, the pages change.  The current build process ensures that the client website and the taskforest.com website stay in sync.  Now if I use WordPress, I don’t want to manually edit the pages using the WP admin site.  I need to install a new plugin that handles inclusion of files.  So, I installed the WP Include plugin.  I’d have to change my build process, but I could get it to work.

Putting It All Together

Okay, so I got the themes and plugins installed, and I’ve got the process worked out.  It was then time to try a proof of concept with a single page.  It worked just as expected, but the site seemed very sluggish.  I made sure I didn’t have anything enabled that I didn’t need.  Still, the site was noticeably slower than the existing site, and that’s with only one non-blank page and zero blog posts.  I thought that maybe the problem was that the bluehost.com shared server was too slow.  I just happened to have an unused server in the same data center that hosts taskforest.com.  The same kind of server as well. It took the better part of the morning, but I installed PHP, mysql and WordPress on that server.  In installed PHP as a static module within Apache, for optimal performance.  Even then, on a pristine machine running nothing else, it was slow.

How Slow is Slow?

Anyone who knows me knows that I’m an Engineer, and as an Engineer I like metrics.  I wanted to know that it wasn’t just my own bias that was penalizing WordPress over my established way of doing things.  So, I looked at the Safari web browser on the Mac.  Safari has a really useful feature called the Web Inspector that, among other things, displays the amount of time it takes for different parts of a page to load.  The numbers were very surprising.  With WordPress, a new page would take 1.75 seconds to load – an eternity on a high-speed broadband connection.  A subsequent request of the same page would take about 700ms. Switching to the TaskForest website’s strategy would take about 800ms for a new page, and 275ms for a cached page.  That’s more than a 200% increase in speed!  The TaskForest website used a preprocessor that allowed for including header files.  It ran under mod_perl.  I decided to write a spider that crawled the website and convert everything to static pages, and serve those pages without mod_perl.  With this enhancement, the results were even better.  I saw a 400% to 800% speed increase, with cached pages taking just 140ms to load.

One of the reasons WordPress’s numbers were so poor is that if articles like this one are still correct, WordPerfect does not cache the content of pages by default.  Resources within the HTML page, like images or javascript files seem to be cached, but every page still does upto 120 database accesses depending on the setup.  Apache is a lot better at caching requests for static HTML pages.  Even the TaskForest webserver used for the REST API, caches data very intelligently using the HTTP headers.

What Does This All Mean?

Given the amount of compromises I would have to make in the design, layout and build process, switching to WordPress would have been very painful.  However, with the performance hit the site would be taking, it seems unlikely that I’ll switch to WP any for the TaskForest website any time soon.  I think I’d rather write a script to generate RSS feeds on demand and automatically submit the feed to feed notification sites like pingomatic. I think this was a great learning experience that will help me with similar decisions in the future.  And I also got to write a cool spider and increase the speed of the TaskForest web site.

Safari's Web Inspector

There are, however, many classes of applications for which text files are the preferred means of storing data.  One of the main reasons is that when data is stored in a relational database, editing it is not a trivial task.  A well-normalized database is not easily updated via an SQL command line.  More often than not, a dedicated, graphical editor is needed to model the complex relationships.

Several years ago, when I wrote TaskForest, one of the initial design requirements was that it be easily configurable with just a shell prompt and one’s favorite text editor. Many of the servers I cared for for schools and non-profits were old boxes which I administered by logging into them via ssh. So when it came to designing job definitions and dependencies, I chose a text file representation. The benefits of text files over a graphical user interface for this include:

Easy Remote Access

All you need is the ability to get to a command line and a text editor on the machine that holds the data files. With the such low client access requirements, virtually any old machine that has internet access and an ssh client can be used to administer the system. I have often worked on my own taskforestd server from a local internet cafe using a Putty.exe downloaded minutes earlier.

Mobile Access

Text files also make work relatively easy using a mobile ssh client like Idokorro Mobile SSH. A dedicated mobile client would be ideal, but short of that, the text file approach assures low bandwidth usage and easy-to-make changes.

Flexibility

The simple, easily parseable format of text files allows us to build richer graphical clients later that would use a graphical interface to specify relationships between jobs.

Source Control

The text based format makes it easy to place the data files under source control. You can also easily diff different versions of the same data file.

Grep

When you have dozens of job group files and hundreds of jobs, you may need to answer questions like: “Are we still running Job J?” This can easily be answered by grepping the files for job J.

Low footprint

When you’re designing an open-source application you may want to minimize the complexity of the system by not forcing dependencies on major subsystems like GUI libraries and Relational Databases.  Of course there is a point at which a such dependencies are inevitable – you have to periodically re-evaluate your decisions and determine whether decisions that were correct, say, a year ago are still correct today.

Choosing A Text Format

In the case of the TaskForest project, the most difficult task by far was choosing which text format to use.  I went through several iterations trying to find one that was simple to read and write, and yet rich enough to model the domain space completely.  What worked for me (and might work for you) was to ask myself how I would represent the data I’m trying to save given just a pencil and paper.  Drawing in a notebook gave me the flexibility to sketch and edit easily, and once I had a good representation, converting that to a text file was a simple task.

Streams at the Great Smoky Mountains

The Common Weakness Enumeration (CWE) has released their list of Top 25 Most Dangerous Programming Errors. This list and the explanations of the errors are very instructive and should help both novice and expert programmers.  If you’re a developer, I strongly urge you to read this document and make sure you understand the concepts it covers.