As I was beginning to lose hope of finding a usable app, and second-guess my choice of phones, I found Gentle Alarm by mobitobi.   At €1.59 it’s one of the best-written Android applications I’ve bought so far.

It’s got the fundamentals right – multiple alarms (sorted by time of day!), snooze, customizable sounds, etc.  It’s also got a feature I didn’t think I wanted, but now need.  The writers of the software believe that when you wake up during the REM phase of your sleep (light sleep), you’re more refreshed than when you wake up from deep sleep.  To that end, if you want you can create alarm profiles where the alarms start ringing very softly as much as 30 minutes prior to the actual alarm time.

The idea is that if you’re in REM sleep, the soft alarm will be enough to wake you.  If you’re in deep sleep, you’ll sleep right through it, and hopefully be back in REM by the main alarm time.  So really what you’re doing is setting two alarms for each alarm time – the pre-alarm that’s really soft and starts a configurable amount of time prior to the second alarm – the one that’s louder.  Even the main alarm can be configured so that the volume gently gets louder over a period of time.

I’ve found this gradual ramping-up of intensity to be very comforting – I no longer have to be shocked out of bed by a blaring buzz next to my ear.

The app also has other features that I don’t use like a captcha system to ‘prove you’re awake’ so that you don’t turn the alarm off in your sleep.  It also has a night-time display that can be muted so that it doesn’t illuminate a dark bedroom.

There are several wristwatches and bedside alarm clocks available that provide this functionality, but they cost hundreds of dollars (US).  I’ve found that this app gets the job done admirably for a fraction of the cost.  There is one thing you have to be careful about: Because of the way Android handles long-running tasks you have to make sure that you’ve installed a task manager like “Advanced Task Manager” and that Gentle Alarm is on its “exception list.”  This way, the system won’t kill the Alarm Clock process thinking it’s a long-running job that’s needlessly using system resources.

Under the tracks at Washington and Canal

When you’re working in Unix or Linux or even Mac OS X, there are often times when you need to apply the same command to a list of files.  In this post I’ll show you a couple of quick ways to do this using the bash shell.

Let’s assume you have a command called process that takes a file name as its input and processes the file in place.  For example, if you want to process the file at /tmp/f1, the command would be

process /tmp/f1

.  Let’s also say that after processing a file you want to archive it, using the archive command.

Now, if you want to process and archive files /tmp/f1, /tmp/f2 and /tmp/f3 you could either do this:

./process /tmp/f1
./archive /tmp/f1
./process /tmp/f2
./archive /tmp/f2
./process /tmp/f3
./archive /tmp/f3

or, you could use the bash keyword for and process all files in a loop like this:

for f in /tmp/f1 /tmp/f2 /tmp/f3
do
    ./process $f
    ./archive $f
done

That’s a lot less typing.  Let’s look at this loop closely:

A for loop used like this has three key components: the loop variable, the loop list and the loop body.  Let’s identify them in reverse order: The loop body is the part between the do and done keywords.  The loop list is the list of ‘things’ over which you want the for loop to iterate.  In this case, the loop list is the list of files /tmp/f1 through /tmp/f3.  Note that the items in the loop list are separated by spaces.

The last component that makes the for loop work is the loop variable.  In this example the loop variable is called f.  This means that in the body of the loop, the loop variable f will be used as a placeholder for each of the items in the loop list.

The first time through the loop, the loop variable f will have the value /tmp/f1.  To refer to the value of the loop variable, you put a ‘$’ before its name.  That’s why the body of the loop contains process $f and not process f.  process f would attempt to process a file named ‘f’, while process $f processes /tmp/f1 (in the first iteration of the loop).

Let’s test this out just to be sure.  Let’s create a dummy process program that does nothing other than print out the name of the file it’s processing.

#!/bin/bash

echo "process - processing file $1";

Let’s also create an archive program that prints out the name of the file it’s archiving:

#!/bin/bash

echo "archive - archiving file $1";

Now, you could type the following on the command line:

for f in /tmp/f1 /tmp/f2 /tmp/f3
do
    ./process $f
    ./archive $f
done

but if you’re like me and you prefer to see everything on the same line, you could instead type in the following one line:

for f in /tmp/f1 /tmp/f2 /tmp/f3; do ./process $f; ./archive $f; done

What’s different here is that you need the two semicolons (;) to tell bash where the loop list and loop body end.

Whichever method you choose, you’ll see the following output:

process - processing file /tmp/f1
archive - archiving file /tmp/f1
process - processing file /tmp/f2
archive - archiving file /tmp/f2
process - processing file /tmp/f3
archive - archiving file /tmp/f3

As you can see, the loop body was executed three times, and each time the value of the loop variable f was used wherever there was a $f in the loop body.

Using Wildcards With Loops

Sometimes you may have so many files in the loop list that you don’t want to type all their names onto the comand line.  In cases like this you could use wildcards.

Let’s say you want to process all files whose names end with .txt. You can do that by replacing the loop list with the appropriate regular expression:

for f in *.txt; do ./process $f; ./archive $f; done

Now, even if you have a hundred .txt files to process, this short command line will still work for you.

If you want to process all .txt and .jpg files, you could use either of the following two methods:

for f in *.txt *.jpg ; do ./process $f; ./archive $f; done

or

for f in *.{txt,jpg}; do ./process $f; ./archive $f; done

In Conclusion

As you can see, it’s easy to perform the same operations on a bunch of files when you use loops.  Loops are exceptionally useful as one-time command-line hacks.  It is important to remember that loops are not appropriate for every task.  There are times when the loop body is so complex that it’s safer to abandon the command line method and instead ‘factor out’ the complexity into a script in bash or other scripting language.

There are also times when the files on which you want to operate are all over the disk drive, or in many subdirectories of the current directory.  As we’ll see in a later post, there are other common programs like ‘find’ that are perfect in this case.

At the Agra Fort, on the way to the Public Hall

In a earlier post I wrote about how important it is to have your data backed up.  On my Macs, my main backup utility is Time Machine, which comes pre-installed with the Mac OS.  Time Machine can also back up external hard drives, even though it may not be obvious how to do it.  This article shows you how to change the default settings to do this.

If you don’t have Time Machine – for example, if you’re using Linux -  the rsync command can be almost as easy to use and it can be used to back up your data.  Now, rsync is a very powerful command that has dozens of options, but three options are all you need for simple backups.  Let’s say you save all your photos on an external drive visible at /Volumes/Media.      And  you want to back it up on another drive located at /Volumes/Backup.  The following command would do this for you:

rsync -avh /Volumes/Media /Volumes/Backup

The preceding command will copy all of /Volumes/Media, and place it inside a subdirectory of /Volumes/Backup, called Media.  In other words, /Volume/Backup/Media will contain everything that /Volumes/Media contains.  But here’s the really neat part: Every subsequent time you invoke this command, rsync will only copy those files that are different from what is already in the destination directory (or those files that are new).  So if you copy 500 pictures on one day, and then back up the directory as shown above, and then import 10 new pictures into /Volumes/Media, and then back up again, the second backup will only copy the 10 new pictures, not all 510.

Since Mac OS X is also based on UNIX, the above command will work on the Mac OS just as well as it works on Linux or UNIX.

For more details on the rsync command, please see the rsync documentation page.

Google reported yesterday that their search engine will now include a website’s speed in the list of factors it uses to decide how high to rank the site in its search results.  In this post I consider what this means for web developers and what steps you can take to make your site faster.

Google has made it clear that speed is not the primary factor when determining page rank:

While site speed is a new signal, it doesn’t carry as much weight as the relevance of a page. Currently, fewer than 1% of search queries are affected by the site speed signal in our implementation and the signal for site speed only applies for visitors searching in English on Google.com at this point.

It is, however, a turning point for web site developers.  What Google is really doing is putting a tangible value on the User Experience.  A fast site makes for a more satisfactory experience, and linking to fast sites also makes users happier with Google.  I know how frustrating it is to see what appears to be a promising search result on Google, only to find the site very sluggish once I click through Google’s recommendation.  So, like it or not, you and I will now have to make sure we at least consider our sites’ rendering speeds.

As I had mentioned in a earlier post, I recently changed the taskforest.com website to static HTML from a template-based system.   After reading Google’s blog entry, I installed Firebug on Firefox, as well as two different page performance measuring tools: PageSpeed and YSlow from Yahoo and started testing out the TaskForest web site.  To my dismay, the site scored a paltry 76/100 on PageSpeed and a ‘D’ (67/100) on YSlow.  I’m glad to say that in less than an hour I was able to make the changes that most significantly improved my site’s speed.

There were four major areas in which the two tools recommended changes:

Exploiting Browsers’ Caches

The site was not using the Expires response headers. As this document from Google shows, proper usage of this HTTP header can greatly reduce load on the server as well as clients.The Expires header tells the client how long the server expects the resource to stay unchanged.  For images like logos and screenshots that don’t change very often, this period could (and should) be as high as few months, possibly even the maximum allowed value – one year.  For HTML pages that change often, you could set it to 2-7 days.

The taskforest site which was using Apache 1.41, was not sending the Expires HTTP response header. The reason is that the module that sets this, mod_expires, is not enabled by default.

Combining External CSS

The website was using three CSS style sheets.  One was my own ‘styles.css,’ but the other two were Yahoos YUI style sheets.  For every request for a web page, there were 3 additional requests for style sheets.

Combining External Javascript

Just like style sheets, There were 6 Javascript files that were being served with every page.

Using Compression

The website was not compressing the data it sent to the client.  As this article and the video below from Google shows, data compression can have a major impact on the size of the payload the server delivers to the client, thus making the site faster.

Optimizing the site

So the first order of business was to enable mod-expires and install and activate mod_gzip.  These are the Apache modules that enable the Expires header and Response compression.  I reconfigured Apache with the –enable-module=expires and –activate-module=src/modules/gzip/mod_gzip.c options.  Previously, the relevant lines in my httpd.conf file looked like this:

<VirtualHost 216.139.227.47>
 ServerName www.taskforest.com
 DocumentRoot ".../taskforest/htdocs/website"
</VirtualHost>

After installing the two new modules, my httpd.conf file looked like this:

<VirtualHost 216.139.228.44>
 ServerName www.taskforest.com
 DocumentRoot ".../taskforest/htdocs/website"

 FileEtag None

 # enable expirations
 ExpiresActive On

 # unless overridden elsewhere resources expires
 # after 60 days in the client's cache
 #
 ExpiresDefault "access plus 60 days"

 # The site's feed expires after 1 day in the cache
 ExpiresByType application/xml A86400

 # HTML files expire after 1 day in the cache
 ExpiresByType text/html A86400

 # Turn compressing on
 mod_gzip_on Yes

 # Don't bother compressing tiny files
 mod_gzip_minimum_file_size  1002

 mod_gzip_maximum_file_size  0
 mod_gzip_maximum_inmem_size 60000

 # Compress XML files, text files (including HTML)
 # and directory listings
 mod_gzip_item_include mime "application/xml"
 mod_gzip_item_include mime text/*
 mod_gzip_item_include mime "httpd/unix-directory"

 # Compress files based on their names as well
 mod_gzip_item_include file "\.txt$"
 mod_gzip_item_include file "\.html$"
 mod_gzip_item_include file "\.css$"
 mod_gzip_item_include file "\.js$"

 mod_gzip_dechunk Yes
 mod_gzip_temp_dir "/tmp"
 mod_gzip_keep_workfiles No

</VirtualHost>
<VirtualHost 216.139.228.44>
 ServerName taskforest.com
 RedirectMatch (.*) http://www.taskforest.com$1
</VirtualHost>

For the multiple CSS files, I just copied both of the YUI  style sheets into my styles.css file, preserving the copyright comments.  As for the YUI javascript files, it turns out that none of them were being used!  They were just copied and pasted from the taskforestd webserver web site, which does use the javascript files.  I was able to delete them altogether.  It was rather embarrassing to realize that the site was serving unused files.  But I was glad to have the tools that helped me realize this.

Results of Optimization

The results of the optimization were startling.  The PageTest scores originally looked like this:

PageSpeed Results Before Optimization

Now, after the optimization, the report was much more respectable:

PageSpeed Results After Optimization

This exercise gave me new insight into what is needed to make a website faster.  It’s not just the fastest machine and the most optimized database that matters.  One also has to consider the fundamental characteristics of the Web, and HTTP in particular, where a little effort can bring significant improvement.

In the case of ‘im,’ I first manually went to the imdb.com site and looked for the search form at the top of the page.  I right-clicked on the input field, and selected ‘im’ as the keyword for that search.  So now, whenever I want to look up something on imdb, all I have to do is hit ‘Cmd-L’ on my Mac to shift focus to the URL bar, enter ‘im’ followed by the query string and then hit enter.  Then I’m taken directly to the search results page.  It saves me from having to navigate to imdb.com first. It works really well for all kinds of searches, including CPAN, Wikipedia and Weather.com.

I can’t believe I wasn’t aware of this shortcut all this time

Adding a Keyword to the CPAN Search Box

Entering the Keyword

The Keyword Search In Action

Also that week, many of us heard of Line2, an app for the iPhone that allows you to make calls using WiFi and bypassing AT&T.  After New York Times columnist David Pogue wrote about it, hundreds of readers went to the iTunes store to download the app, only to find it missing.  Soon after, Pogue received hundreds of angry emails blaming him for bringing the app to Apples attention, saying that Apple pulled the app.  After some investigative work Pogue found out that it was the app developers themselves who pulled the app off the iTunes store in response to a denial of service attack – automated sign-ups.  In that article’s comments, Pogue noted that a lot of the finger pointing could have been averted if the developers simply mentioned on their website that they had pulled the app while they worked on the denial of service attack.

These episodes got me thinking about the importance of communicating with clients during a crisis.  Most individuals and businesses that I’m familiar with have very well-planned procedures in the case of fire or building evacuation.  But that’s seldom the case with digital crises like internet access outages or denial of service attacks.

This article on ehow.com has some good instructions on how to communicate during crises.  My own advice is to pay attention to the key questions:

How

Have a plan that specifies how you will communicate with clients, especially when your normal means of communication are not available.  In the case of my friend, he couldn’t email his clients, or even update his own website.  I think the best thing he could have done to prepare for this would be to have one name server at a competitor’s data center, and have that name server be one of those responsible for his domain.  As his clients noticed their own sites were down, the next thing most of them would do (as I did) is check the website of the data center.  With a couple of quick commands, he could redirect his domain to a static web page that would give clients the information they need.  If that were planned in advance, this entire setup could be arranged for about $6 to $10 per month.  Not bad at all.

Who

Identify what needs to be done to open the channels of communication during a crisis and assign your staff specific roles.  If you’re the president of the company you may be too busy working behind the scenes calling in favors or negotiating with counterparties.  Again, in my friend’s case, while he’s on the phone calling his vendors, a member of his staff could be  running to the local Starbucks coffee shop with a laptop and logging in to the name server on their competitor’s network and changing their DNS records. And then staying there for updates that need to be relayed to the clients.

When

During a crisis, the most stressful thing for clients is ‘radio silence.’  Sometimes it’s inevitable that there will be nothing new to report for long periods of time.  Even then, let your clients know that there is nothing new to report.  Depending on the business, and the nature of the crisis,  the frequency of updates could be anywhere from five minutes to a few hours.

What

This is really important.  Tell the truth.  Clients will bolt if they lose their trust in you.  If you tell them the truth they will likely sympathize with your situation and even offer to help.  Oh, and telling the truth is also the Right Thing To Do.  A 36 hour outage for a data center is usually catastrophic.  I was sure my friend was going to lose some big clients after such a prolonged outage.  I called him a few days after the dust settled and asked him about the fallout.  There was none.  He told everyone the truth – whether they paid $20 a month or thousands of dollars a month.  His biggest clients physically came in to the data center and weathered the storm with him.  They knew exactly what was going on and offered to help in many tangible ways.  Even though, in my opinion, my friend didn’t do a good job on the “How,” “Who,” and “When,” telling the truth to everyone who was able to get in touch with him allowed him to emerge from this crisis relatively unscathed.

Setting up a Sandbox Domain

The first step in trying out WordPress was to set up a new domain just for testing out the WordPress installation.  This way, I wouldn’t affect the taskforest.com domain during my experiments.  I happen to run my own name servers using Daniel Bernstein’s tinydns, so I decided to create a new domain called tf.enoor.com, a subdomain of my defunct company’s domain.  Since I use bluehost.com’s WordPress hosting service, I had to make their name servers responsible for the domain.  All that’s needed is adding the following two lines to /etc/tinydns/root/data:

&tf.enoor.com:74.220.195.31:ns1.bluehost.com:300
&tf.enoor.com:69.89.16.4:ns2.bluehost.com:300

Selecting a Theme

After setting up WordPress, the next step was to select a theme.  I wanted something similar to TaskForest’s current design, so I chose the remarkably customizable Suffusion theme.  After some tinkering I was able to get a site that was quite similar to the original, with one compromise – I could get either the logo or the site’s name in the header, but not both.  Not the way they are on the current site.  It’s quite important that the site’s name appear in the header, because that helps with Search Engine Ranking.  So with a heavy heart I decided to omit the logo.

The Problem with Content

TaskForest ships with its own web server to support REST interface.  As part of the included website is all the documentation for the system. Just like the code, this documentation is under source control, and it’s also used to populate the taskforest.com website.  This way, I can ensure that both the taskforest.com website and a user’s local install have the most up-to-date docs, as long as the user is running the latest version of the software.  What this also means is that I have a few dozen webpages that need to be transfered to WordPress before the new site can go live.  I was already resigned to the fact that the URLs of these pages would be different – the current site has URLs that look like http://www.taskforest.com/about.html, but the default WordPress installation would use URLs that look like http://www.taskforest.com/about/.  It’s not a huge deal, but I prefer my way.

The bigger issue is that when a new version of the software is released, the pages change.  The current build process ensures that the client website and the taskforest.com website stay in sync.  Now if I use WordPress, I don’t want to manually edit the pages using the WP admin site.  I need to install a new plugin that handles inclusion of files.  So, I installed the WP Include plugin.  I’d have to change my build process, but I could get it to work.

Putting It All Together

Okay, so I got the themes and plugins installed, and I’ve got the process worked out.  It was then time to try a proof of concept with a single page.  It worked just as expected, but the site seemed very sluggish.  I made sure I didn’t have anything enabled that I didn’t need.  Still, the site was noticeably slower than the existing site, and that’s with only one non-blank page and zero blog posts.  I thought that maybe the problem was that the bluehost.com shared server was too slow.  I just happened to have an unused server in the same data center that hosts taskforest.com.  The same kind of server as well. It took the better part of the morning, but I installed PHP, mysql and WordPress on that server.  In installed PHP as a static module within Apache, for optimal performance.  Even then, on a pristine machine running nothing else, it was slow.

How Slow is Slow?

Anyone who knows me knows that I’m an Engineer, and as an Engineer I like metrics.  I wanted to know that it wasn’t just my own bias that was penalizing WordPress over my established way of doing things.  So, I looked at the Safari web browser on the Mac.  Safari has a really useful feature called the Web Inspector that, among other things, displays the amount of time it takes for different parts of a page to load.  The numbers were very surprising.  With WordPress, a new page would take 1.75 seconds to load – an eternity on a high-speed broadband connection.  A subsequent request of the same page would take about 700ms. Switching to the TaskForest website’s strategy would take about 800ms for a new page, and 275ms for a cached page.  That’s more than a 200% increase in speed!  The TaskForest website used a preprocessor that allowed for including header files.  It ran under mod_perl.  I decided to write a spider that crawled the website and convert everything to static pages, and serve those pages without mod_perl.  With this enhancement, the results were even better.  I saw a 400% to 800% speed increase, with cached pages taking just 140ms to load.

One of the reasons WordPress’s numbers were so poor is that if articles like this one are still correct, WordPerfect does not cache the content of pages by default.  Resources within the HTML page, like images or javascript files seem to be cached, but every page still does upto 120 database accesses depending on the setup.  Apache is a lot better at caching requests for static HTML pages.  Even the TaskForest webserver used for the REST API, caches data very intelligently using the HTTP headers.

What Does This All Mean?

Given the amount of compromises I would have to make in the design, layout and build process, switching to WordPress would have been very painful.  However, with the performance hit the site would be taking, it seems unlikely that I’ll switch to WP any for the TaskForest website any time soon.  I think I’d rather write a script to generate RSS feeds on demand and automatically submit the feed to feed notification sites like pingomatic. I think this was a great learning experience that will help me with similar decisions in the future.  And I also got to write a cool spider and increase the speed of the TaskForest web site.

Safari's Web Inspector

As Bob points out, when you submit a photograph to Frommer’s  not only do they want the rights to use it for the stated purpose, they also want all rights to do anything with your picture, even license it out to others for perpetuity.  In other words, they are asking for the rights of the photographer, without paying for what those rights are worth.  This is regardless of whether a photo is accepted for publication or wins a contest or not.  It appears that they’re counting on the fact that most people who submit photos will not read the fine print, or will be amateurs who are just happy that they have a picture good enough to send to a competition.  As Bob correctly points out, this kind of action will end up ‘kill[ing] the freelancers’ who make their livelihoods selling the rights to use their pictures, not all the rights to their pictures.

I went to their website and entered the following on their ‘Contact Us’ page:

This is is regard to the issues brought up at Bob Krist’s blog, found at http://www.bobkrist.com/blog/frommers-now-fing-freelancers-and-photo-enthusiasts/

I’m not a professional photographer but I am a frequent traveler.  I just want you to know that I find your company’s take on photographers’ rights reprehensible, and that I intend to boycott you and your products, and urge all my friends and family to do the same – until you change your policy on seeking perpetual rights for submitted photographs.

Please reconsider your position.

Sincerely,

Aijaz Ansari.

Keeping true to my word, I urge you, readers of this blog, to boycott Frommer’s products in protest of their licensing policies.

A Stop Sign in Sharjah, U.A.E.