There are, however, many classes of applications for which text files are the preferred means of storing data.  One of the main reasons is that when data is stored in a relational database, editing it is not a trivial task.  A well-normalized database is not easily updated via an SQL command line.  More often than not, a dedicated, graphical editor is needed to model the complex relationships.

Several years ago, when I wrote TaskForest, one of the initial design requirements was that it be easily configurable with just a shell prompt and one’s favorite text editor. Many of the servers I cared for for schools and non-profits were old boxes which I administered by logging into them via ssh. So when it came to designing job definitions and dependencies, I chose a text file representation. The benefits of text files over a graphical user interface for this include:

Easy Remote Access

All you need is the ability to get to a command line and a text editor on the machine that holds the data files. With the such low client access requirements, virtually any old machine that has internet access and an ssh client can be used to administer the system. I have often worked on my own taskforestd server from a local internet cafe using a Putty.exe downloaded minutes earlier.

Mobile Access

Text files also make work relatively easy using a mobile ssh client like Idokorro Mobile SSH. A dedicated mobile client would be ideal, but short of that, the text file approach assures low bandwidth usage and easy-to-make changes.

Flexibility

The simple, easily parseable format of text files allows us to build richer graphical clients later that would use a graphical interface to specify relationships between jobs.

Source Control

The text based format makes it easy to place the data files under source control. You can also easily diff different versions of the same data file.

Grep

When you have dozens of job group files and hundreds of jobs, you may need to answer questions like: “Are we still running Job J?” This can easily be answered by grepping the files for job J.

Low footprint

When you’re designing an open-source application you may want to minimize the complexity of the system by not forcing dependencies on major subsystems like GUI libraries and Relational Databases.  Of course there is a point at which a such dependencies are inevitable – you have to periodically re-evaluate your decisions and determine whether decisions that were correct, say, a year ago are still correct today.

Choosing A Text Format

In the case of the TaskForest project, the most difficult task by far was choosing which text format to use.  I went through several iterations trying to find one that was simple to read and write, and yet rich enough to model the domain space completely.  What worked for me (and might work for you) was to ask myself how I would represent the data I’m trying to save given just a pencil and paper.  Drawing in a notebook gave me the flexibility to sketch and edit easily, and once I had a good representation, converting that to a text file was a simple task.

190 S. La Salle St. - Downtown Chicago

The #grid website has a great tool for web designers -it “inserts a layout grid in web pages, allows you to hold it in place, and toggle between displaying it in the foreground or background.”  Go to their website and have a look.  It’s pretty impressive.  Simple, but impressive.  I think I’m gonna give this a shot for the next web site I design.  I think it would be really useful in development, not as much in a production environment.

Streams at the Great Smoky Mountains

The Common Weakness Enumeration (CWE) has released their list of Top 25 Most Dangerous Programming Errors. This list and the explanations of the errors are very instructive and should help both novice and expert programmers.  If you’re a developer, I strongly urge you to read this document and make sure you understand the concepts it covers.

The Tautline Hitch

It is very common to want a taut line (rope) between two objects.  In these cases it is not enough to just connect a rope to the two endpoints.  The rope must also stay stretched and not sag.  If you’ve ever had to tie a volleyball net or a tent guy line to a stake, you know difficult it is to tie the knot at just the right place to keep the line tight.  It doesn’t help when the line sags every so often and needs to be retightened.

The tautline hitch knot is perfect for this application.  It makes a loop in the rope and the position of the knot affects length of the loop.  The more you move the knot towards the opposite end of the rope, the tighter the line gets.  The knot is like a ratchet.  It can be slid up and down the rope, but it does not slide when the rope is under tension.  It only moves when you hold the knot and push it one way or the other.  This way when a rope sags, you can hold the knot and move it from one endpoint to the other without having to untie the knot and retie it.

There are many resources out there for learning how to tie knots.  My favorites are the Klutz Book of Knots, The Complete Book of Knots, and the Animated Knots website.  Of course, no discussion on knots can be complete without mentioning the definitive reference, Ashley Book of Knots, with more than 3,900 different knots and 7,000 illustrations.

$ pwd
/Users/aijaz
$ cd Documents/Training/qt/ch1/
$ ls
p1    t2
$

$ cd ~/Projects/projects/taskforest/lib/TaskForest/
$ ls -l
total 352
-rw-r--r--   1 aijaz  aijaz   8816 May 25  2009 Calendar.pm
...
-rw-r--r--   1 aijaz  aijaz   4558 May 25  2009 TimeDependency.pm
$

$ cd ~/tc/config/rates/
$ ls -l
total 27544
drwxr-xr-x  21 aijaz  aijaz       714 Sep  3  2007 Text
...
-rwxr-xr-x   1 aijaz  aijaz       569 Sep  3  2007 findVoipAccess.pl
$

$ d
 0  ~/tc/config/rates
 1  ~/Projects/projects/taskforest/lib/TaskForest
 2  ~/Documents/Training/qt/ch1
 3  ~
#: 2
$ pwd
/Users/aijaz/Documents/Training/qt/ch1
$ d
 0  ~/Documents/Training/qt/ch1
 1  ~/tc/config/rates
 2  ~/Projects/projects/taskforest/lib/TaskForest
 3  ~
#: q
$

$ p
$ pwd
/Users/aijaz/tc/config/rates
$ d
 0  ~/tc/config/rates
 1  ~/Projects/projects/taskforest/lib/TaskForest
 2  ~
#: q
$

Now let’s have a look at the code that makes this work. You can copy and paste this code directly into your .bashrc file.

# An enhanced 'cd' - push directories
# onto a stack as you navigate to it.
#
# The current directory is at the top
# of the stack.
#
function stack_cd {
    if [ $1 ]; then
        # use the pushd bash command to push the directory
        # to the top of the stack, and enter that directory
        pushd "$1" > /dev/null
    else
        # the normal cd behavior is to enter $HOME if no
        # arguments are specified
        pushd $HOME > /dev/null
    fi
}
# the cd command is now an alias to the stack_cd function
#
alias cd=stack_cd

# Swap the top two directories on the stack
#
function swap {
    pushd > /dev/null
}
# s is an alias to the swap function
alias s=swap

# Pop the top (current) directory off the stack
# and move to the next directory
#
function pop_stack {
    popd > /dev/null
}
alias p=pop_stack

# Display the stack of directories and prompt
# the user for an entry.
#
# If the user enters 'p', pop the stack.
# If the user enters a number, move that
# directory to the top of the stack
# If the user enters 'q', don't do anything.
#
function display_stack
{
    dirs -v
    echo -n "#: "
    read dir
    if [[ $dir = 'p' ]]; then
        pushd > /dev/null
    elif [[ $dir != 'q' ]]; then
        d=$(dirs -l +$dir);
        popd +$dir > /dev/null
        pushd "$d" > /dev/null
    fi
}
alias d=display_stack

I am certain Google tested their application thoroughly.  They’ve been known to do extensive usability tests for the seemingly tiniest of changes to their web site.  But even the most well-implemented tests are incomplete if they’re not performed on a statistically representative sample of the audience.

But today, the BBC reported that Google has admitted that they only tested Buzz internally, and bypassed their regular rigorous testing procedures — possibly in an attempt to get it out the door as soon as possible. I’ll let the pundits decide if it did more harm than good to the firm, but it’s a warning to other software developers: skipping testing can lead to embarrassing failures.

Google Buzz

Almost immediately after Google started rolling out Buzz to its Gmail clients various blogs reported that there was an egregious flaw in the system that exposed people’s private contacts to the rest of the world.  When a Gmail user logged in, they were presented with a dialog box that asked them if they wanted an introduction to Buzz or not.  If the user declined, it didn’t mean that Buzz was disabled – it only meant that they didn’t want the intro.  What Google did behind the scenes was that they scanned the user’s emails and collected the emails of the people with whom the user corresponded most often.  Then, it automatically modified the user’s Gmail account to ‘follow’ those people in Buzz.

But here’s the thing that infuriated most people – it made that list of the people that user’s following public: Anyone who looked at the user’s public Google profile could see who that person was following.  In other words, anyone looking at that user’s public profile could see who that user emailed most often.

Now pretend you’re a reporter who’s communicating with dissidents in Myanmar or Iran.  Would you want your heretofore anonymous sources made public?  In one reported case, a woman’s most frequent contacts were her boyfriend and her mother.  Her third-most-frequent contact (with a lot fewer emails) was her abusive ex-husband, from whom she had kept her email address hidden.  But what Google did when they released Buzz is automatically have her follow her ex, and share her shared Google Reader messages, including her location and workplace with her ex.

Another blogger reported that her Android phone uploaded to her public profile a photo that she had taken, but not uploaded. She thought her photo was private, but Google’s assumption was that if the photo was on her phone, it would be okay to upload it to her public profile.

The third problem that surfaced in the two days after Buzz was released was that in some cases users thought that they were referencing a person’s public email address in a buzz (think of it as a retweet or a reply) but in reality were referencing a private email address.  When they selected a person by their name, it wasn’t clear which email address was linked to that name.  For example, if John Smith’s public email address is john_smith@example.com, and his personal address is js_123@example.com, and the public address is associated with a Google profile, a Buzz user may inadvertently select the private email address in a buzz, thus exposing John’s private address to everyone he follows.

The iPad

In the minutes after Apple’s Steve Jobs announced the new mobile appliance’s name, people started making very funny, original and sometimes vulgar jokes about the name iPad.  I could post them here, but the jokes are old by now.

Salacious Texts On Samsung Virgin Mobile Phones

According to this article, a 49 year old man from Winnipeg can no longer have a relationship with the woman he loved because she accused him of cheating on him.  Her proof?  Sexy text messages on his cell phones.  He denied the affair, but couldn’t explain how they got there.  Only later did he learn that the messages were pre-loaded by Virgin Mobile.

What Do These Cases Have in Common?

As an outsider it appears to me that in each of the three cases the problem was that the population for whom the system was designed wasn’t necessarily the only population actually using the system.  Let’s look at the Google Buzz case.  If we claim that Google is neither evil nor stupid, what went wrong?  I think there were several assumptions that the Google developers and designers made that were incorrect.

Buzz seemed to have be designed for people who collaborate together in tightly knit groups.  The assumption is that people who want to share information with each other communicate a lot with each other.  In real life that isn’t always true, and neither is the converse.  In many cases the people that I want to share the most with are people with whom I have so many channels of communication (including the dinner table) that any single channel like email contains a fraction of the actual communication taking place.  Similarly, as in the case of the lady who was hiding her online identity from her ex, the people we communicate with aren’t always the people we want to share information with.  If Google’s designers approached their product with that bias, they could have made (and did make) a technically excellent product that worked as they intended, and whose test cases passed, but which did exactly the wrong thing.

I think that that’s essentially the core problem with the issue of the publication of private photos and email addresses.  The joke in the blogs was that Buzz was designed for an people like engineers at Google, not for people who go to parties and take pictures that they would regret the next morning (and I know they’re not mutually exclusive groups).  Unlike in the business world and the networks of sites like LinkedIn, people sometimes guard their email addresses jealously and do not want them published without their authorization.

I am certain Google tested their application thoroughly.  They’ve been known to do extensive usability tests for the seemingly tiniest of changes to their web site.  But even the most well-implemented tests are incomplete if they’re not performed on a statistically representative sample of the audience.

The same can be said for Apple and Virgin Mobile.  On of the most insightful questions asked after the iPad announcement was, “Did Apple not have any women on the product naming committee?”  Did Virgin Mobile think that their only consumers would be teenagers who would find the default text messages cute and funny and have the technical expertise to realize that these messages were pre-loaded?  While Apple seemed to have failed to consider about half of the human race while naming their product, Virgin failed to consider the less savvy part of the population who are their customers.

What I don’t know is this: Let’s assume I’m wrong, and that in all three cases a statistically-representative portion of the population was tested or considered.  If that’s true, then could these three cases be considered edge conditions?  While a problem could exist in theory could the actual problems be so few that they’re statistically insignificant?  If you’re the person whose lost a relationship or now have to worry about being stalked by an ex it doesn’t really matter.  It’s affected your life in a significant way.

What Can Software Developers Learn From This?

I don’t think anyone or any company (not even Google) can make the perfect application that pleases everyone.  But I think that there are a few rules of thumb that we should follow, to prevent embarrassing and distressing problems.

First: Applications like these should not take action on behalf of the user.  The system should assume that the user always knows what’s best for them.  The most the program should do is make suggestions that the user can then decide to obey or ignore.  This is exactly what Google did (among other things) to change Buzz after the huge public outcry.

Second: Features, no matter how well-intentioned, no should never be opt-out.  They should always be opt-in.  Buzz was essentially an opt-out system.  You could be vulnerable by not doing anything. It isn’t that way now, but it was for those two days when Google’s reputation took a massive hit.

Third: Designers and service providers need to pay careful attention to who their users actually are.  They’re not necessarily men, not necessarily tech-savvy teens or professionals looking to increase their network.  They could include people very different from us.

As developers and designers we need to realize that the products we make will be used in manners that we never imagined, by people who we don’t understand.  If we give our customers the ability to control how our products and services are used, we’ll be helping them and thus helping ourselves.

This got me thinking about the benefits of using third-party libraries in our own code.  Why should a program depend on so many external modules when it could implement what it needs itself?

The answer, in one word, is “Testing.”  A well-written piece of software isn’t considered complete if it doesn’t have well-documented tests that exercise every part of the program.

There are many common tasks that most programs perform – such as reading configuration food, performing date arithmetic and writing to log files.  Every popular language has third-party libraries that perform these tasks correctly and reliably.  If I were to try to duplicate them, it would almost certainly result in inferior code; the mature libraries have had the benefits of extensive testing as well as enhancements and fixes of bugs reported by users.  Using third-party software allows me to focus on my program and what makes it unique.

Perl can be a little stubborn, sometimes.