Under the tracks at Washington and Canal

When you’re working in Unix or Linux or even Mac OS X, there are often times when you need to apply the same command to a list of files.  In this post I’ll show you a couple of quick ways to do this using the bash shell.

Let’s assume you have a command called process that takes a file name as its input and processes the file in place.  For example, if you want to process the file at /tmp/f1, the command would be

process /tmp/f1

.  Let’s also say that after processing a file you want to archive it, using the archive command.

Now, if you want to process and archive files /tmp/f1, /tmp/f2 and /tmp/f3 you could either do this:

./process /tmp/f1
./archive /tmp/f1
./process /tmp/f2
./archive /tmp/f2
./process /tmp/f3
./archive /tmp/f3

or, you could use the bash keyword for and process all files in a loop like this:

for f in /tmp/f1 /tmp/f2 /tmp/f3
do
    ./process $f
    ./archive $f
done

That’s a lot less typing.  Let’s look at this loop closely:

A for loop used like this has three key components: the loop variable, the loop list and the loop body.  Let’s identify them in reverse order: The loop body is the part between the do and done keywords.  The loop list is the list of ‘things’ over which you want the for loop to iterate.  In this case, the loop list is the list of files /tmp/f1 through /tmp/f3.  Note that the items in the loop list are separated by spaces.

The last component that makes the for loop work is the loop variable.  In this example the loop variable is called f.  This means that in the body of the loop, the loop variable f will be used as a placeholder for each of the items in the loop list.

The first time through the loop, the loop variable f will have the value /tmp/f1.  To refer to the value of the loop variable, you put a ‘$’ before its name.  That’s why the body of the loop contains process $f and not process f.  process f would attempt to process a file named ‘f’, while process $f processes /tmp/f1 (in the first iteration of the loop).

Let’s test this out just to be sure.  Let’s create a dummy process program that does nothing other than print out the name of the file it’s processing.

#!/bin/bash

echo "process - processing file $1";

Let’s also create an archive program that prints out the name of the file it’s archiving:

#!/bin/bash

echo "archive - archiving file $1";

Now, you could type the following on the command line:

for f in /tmp/f1 /tmp/f2 /tmp/f3
do
    ./process $f
    ./archive $f
done

but if you’re like me and you prefer to see everything on the same line, you could instead type in the following one line:

for f in /tmp/f1 /tmp/f2 /tmp/f3; do ./process $f; ./archive $f; done

What’s different here is that you need the two semicolons (;) to tell bash where the loop list and loop body end.

Whichever method you choose, you’ll see the following output:

process - processing file /tmp/f1
archive - archiving file /tmp/f1
process - processing file /tmp/f2
archive - archiving file /tmp/f2
process - processing file /tmp/f3
archive - archiving file /tmp/f3

As you can see, the loop body was executed three times, and each time the value of the loop variable f was used wherever there was a $f in the loop body.

Using Wildcards With Loops

Sometimes you may have so many files in the loop list that you don’t want to type all their names onto the comand line.  In cases like this you could use wildcards.

Let’s say you want to process all files whose names end with .txt. You can do that by replacing the loop list with the appropriate regular expression:

for f in *.txt; do ./process $f; ./archive $f; done

Now, even if you have a hundred .txt files to process, this short command line will still work for you.

If you want to process all .txt and .jpg files, you could use either of the following two methods:

for f in *.txt *.jpg ; do ./process $f; ./archive $f; done

or

for f in *.{txt,jpg}; do ./process $f; ./archive $f; done

In Conclusion

As you can see, it’s easy to perform the same operations on a bunch of files when you use loops.  Loops are exceptionally useful as one-time command-line hacks.  It is important to remember that loops are not appropriate for every task.  There are times when the loop body is so complex that it’s safer to abandon the command line method and instead ‘factor out’ the complexity into a script in bash or other scripting language.

There are also times when the files on which you want to operate are all over the disk drive, or in many subdirectories of the current directory.  As we’ll see in a later post, there are other common programs like ‘find’ that are perfect in this case.

In the case of ‘im,’ I first manually went to the imdb.com site and looked for the search form at the top of the page.  I right-clicked on the input field, and selected ‘im’ as the keyword for that search.  So now, whenever I want to look up something on imdb, all I have to do is hit ‘Cmd-L’ on my Mac to shift focus to the URL bar, enter ‘im’ followed by the query string and then hit enter.  Then I’m taken directly to the search results page.  It saves me from having to navigate to imdb.com first. It works really well for all kinds of searches, including CPAN, Wikipedia and Weather.com.

I can’t believe I wasn’t aware of this shortcut all this time

Adding a Keyword to the CPAN Search Box

Entering the Keyword

The Keyword Search In Action

$ pwd
/Users/aijaz
$ cd Documents/Training/qt/ch1/
$ ls
p1    t2
$

$ cd ~/Projects/projects/taskforest/lib/TaskForest/
$ ls -l
total 352
-rw-r--r--   1 aijaz  aijaz   8816 May 25  2009 Calendar.pm
...
-rw-r--r--   1 aijaz  aijaz   4558 May 25  2009 TimeDependency.pm
$

$ cd ~/tc/config/rates/
$ ls -l
total 27544
drwxr-xr-x  21 aijaz  aijaz       714 Sep  3  2007 Text
...
-rwxr-xr-x   1 aijaz  aijaz       569 Sep  3  2007 findVoipAccess.pl
$

$ d
 0  ~/tc/config/rates
 1  ~/Projects/projects/taskforest/lib/TaskForest
 2  ~/Documents/Training/qt/ch1
 3  ~
#: 2
$ pwd
/Users/aijaz/Documents/Training/qt/ch1
$ d
 0  ~/Documents/Training/qt/ch1
 1  ~/tc/config/rates
 2  ~/Projects/projects/taskforest/lib/TaskForest
 3  ~
#: q
$

$ p
$ pwd
/Users/aijaz/tc/config/rates
$ d
 0  ~/tc/config/rates
 1  ~/Projects/projects/taskforest/lib/TaskForest
 2  ~
#: q
$

Now let’s have a look at the code that makes this work. You can copy and paste this code directly into your .bashrc file.

# An enhanced 'cd' - push directories
# onto a stack as you navigate to it.
#
# The current directory is at the top
# of the stack.
#
function stack_cd {
    if [ $1 ]; then
        # use the pushd bash command to push the directory
        # to the top of the stack, and enter that directory
        pushd "$1" > /dev/null
    else
        # the normal cd behavior is to enter $HOME if no
        # arguments are specified
        pushd $HOME > /dev/null
    fi
}
# the cd command is now an alias to the stack_cd function
#
alias cd=stack_cd

# Swap the top two directories on the stack
#
function swap {
    pushd > /dev/null
}
# s is an alias to the swap function
alias s=swap

# Pop the top (current) directory off the stack
# and move to the next directory
#
function pop_stack {
    popd > /dev/null
}
alias p=pop_stack

# Display the stack of directories and prompt
# the user for an entry.
#
# If the user enters 'p', pop the stack.
# If the user enters a number, move that
# directory to the top of the stack
# If the user enters 'q', don't do anything.
#
function display_stack
{
    dirs -v
    echo -n "#: "
    read dir
    if [[ $dir = 'p' ]]; then
        pushd > /dev/null
    elif [[ $dir != 'q' ]]; then
        d=$(dirs -l +$dir);
        popd +$dir > /dev/null
        pushd "$d" > /dev/null
    fi
}
alias d=display_stack

The emacs editor

In their book “The Pragmatic Programmer,” Andrew Hunt and David Thomas stress to “Use a single editor well. The editor should be an extension of your hand; make sure your editor is configurable, extensible, and programmable.”  I couldn’t agree with this more.  A good editor is a very flexible tool.  Once you become an expert at it, you can use it to accomplish tasks almost as quickly as you can think of them.  You need to process a file as a one-off?  You could write a perl or python script, or an awk/sed one-liner, but often you could also do it in an editor – with the benefit of ‘Undo.’

A good editor is configurable, so that you can use it the way that works best for you.  Most editors have more than one way to do things, and there is no right way, just the way that works for you and helps you reach your goal quickly and efficiently. A good editor is extensible, so that it can grow along with your needs.  You need syntax highlighting for the new programming language that was invented after your editor?  You should be able to go online and download an extension module – or write one yourself.  A good editor is programmable.  Your job requires you to document functions and change histories a certain way – your editor should allow you to change its behavior, so that you don’t have to repeat the same kinds of tasks over and over again.

One of the best ways to learn what you can do with editors is to watch others.  Do you have a coworker whose fingers fly across the keyboard while she bends text files to her will? It may not be just that she’s a fast typist – she’s probably really comfortable using her editor.  Search for these people, ask for advice and take notes.  Try different options out, pick an editor, and invest the time to learn it well. You’ll be doing your career a favor.

There is no real correct answer to the question “Which editor is best?”  But there is one absolute: If you work on a Linux or Unix system, you must know how to use the vi editor.  This is because every Linux or Unix system has at least that one editor installed.  Depending on your line of work, it may not be unusual to find yourself on a machine that has only that editor installed.  I started off using vi in college and took the time to learn it well. Years later I found myself in a short lull while switching from the IS41 to the VLR products at Motorola. I took advantage of the downtime by teaching myself emacs and have been primarily an emacs user for since then (about 11 years now). While I prefer emacs, I still use vi every so often, and consider myself fortunate because I know both.

There are many other editors out there and a lot of them would probably meet Hunt’s and Thomas’s criteria, but I would like to add one more to their list: A good editor must be usable without a windowing system. I have also played the role of system administrator, and none of my hosted machines ever had a windowing system installed. This means I could not run eclipse or even jedit. The only editors I could install and use were ones that would work in a text window (an xterm). So it was either vi or emacs.

Pick a good editor, one that meets the criteria listed above, and learn it well. Spend the time to become an expert. The time you invest will be paid back in increased productivity, power and flexibility.

Real Programmers - from http://xkcd.com/378