Under the tracks at Washington and Canal

When you’re working in Unix or Linux or even Mac OS X, there are often times when you need to apply the same command to a list of files.  In this post I’ll show you a couple of quick ways to do this using the bash shell.

Let’s assume you have a command called process that takes a file name as its input and processes the file in place.  For example, if you want to process the file at /tmp/f1, the command would be

process /tmp/f1

.  Let’s also say that after processing a file you want to archive it, using the archive command.

Now, if you want to process and archive files /tmp/f1, /tmp/f2 and /tmp/f3 you could either do this:

./process /tmp/f1
./archive /tmp/f1
./process /tmp/f2
./archive /tmp/f2
./process /tmp/f3
./archive /tmp/f3

or, you could use the bash keyword for and process all files in a loop like this:

for f in /tmp/f1 /tmp/f2 /tmp/f3
do
    ./process $f
    ./archive $f
done

That’s a lot less typing.  Let’s look at this loop closely:

A for loop used like this has three key components: the loop variable, the loop list and the loop body.  Let’s identify them in reverse order: The loop body is the part between the do and done keywords.  The loop list is the list of ‘things’ over which you want the for loop to iterate.  In this case, the loop list is the list of files /tmp/f1 through /tmp/f3.  Note that the items in the loop list are separated by spaces.

The last component that makes the for loop work is the loop variable.  In this example the loop variable is called f.  This means that in the body of the loop, the loop variable f will be used as a placeholder for each of the items in the loop list.

The first time through the loop, the loop variable f will have the value /tmp/f1.  To refer to the value of the loop variable, you put a ‘$’ before its name.  That’s why the body of the loop contains process $f and not process f.  process f would attempt to process a file named ‘f’, while process $f processes /tmp/f1 (in the first iteration of the loop).

Let’s test this out just to be sure.  Let’s create a dummy process program that does nothing other than print out the name of the file it’s processing.

#!/bin/bash

echo "process - processing file $1";

Let’s also create an archive program that prints out the name of the file it’s archiving:

#!/bin/bash

echo "archive - archiving file $1";

Now, you could type the following on the command line:

for f in /tmp/f1 /tmp/f2 /tmp/f3
do
    ./process $f
    ./archive $f
done

but if you’re like me and you prefer to see everything on the same line, you could instead type in the following one line:

for f in /tmp/f1 /tmp/f2 /tmp/f3; do ./process $f; ./archive $f; done

What’s different here is that you need the two semicolons (;) to tell bash where the loop list and loop body end.

Whichever method you choose, you’ll see the following output:

process - processing file /tmp/f1
archive - archiving file /tmp/f1
process - processing file /tmp/f2
archive - archiving file /tmp/f2
process - processing file /tmp/f3
archive - archiving file /tmp/f3

As you can see, the loop body was executed three times, and each time the value of the loop variable f was used wherever there was a $f in the loop body.

Using Wildcards With Loops

Sometimes you may have so many files in the loop list that you don’t want to type all their names onto the comand line.  In cases like this you could use wildcards.

Let’s say you want to process all files whose names end with .txt. You can do that by replacing the loop list with the appropriate regular expression:

for f in *.txt; do ./process $f; ./archive $f; done

Now, even if you have a hundred .txt files to process, this short command line will still work for you.

If you want to process all .txt and .jpg files, you could use either of the following two methods:

for f in *.txt *.jpg ; do ./process $f; ./archive $f; done

or

for f in *.{txt,jpg}; do ./process $f; ./archive $f; done

In Conclusion

As you can see, it’s easy to perform the same operations on a bunch of files when you use loops.  Loops are exceptionally useful as one-time command-line hacks.  It is important to remember that loops are not appropriate for every task.  There are times when the loop body is so complex that it’s safer to abandon the command line method and instead ‘factor out’ the complexity into a script in bash or other scripting language.

There are also times when the files on which you want to operate are all over the disk drive, or in many subdirectories of the current directory.  As we’ll see in a later post, there are other common programs like ‘find’ that are perfect in this case.

$ pwd
/Users/aijaz
$ cd Documents/Training/qt/ch1/
$ ls
p1    t2
$

$ cd ~/Projects/projects/taskforest/lib/TaskForest/
$ ls -l
total 352
-rw-r--r--   1 aijaz  aijaz   8816 May 25  2009 Calendar.pm
...
-rw-r--r--   1 aijaz  aijaz   4558 May 25  2009 TimeDependency.pm
$

$ cd ~/tc/config/rates/
$ ls -l
total 27544
drwxr-xr-x  21 aijaz  aijaz       714 Sep  3  2007 Text
...
-rwxr-xr-x   1 aijaz  aijaz       569 Sep  3  2007 findVoipAccess.pl
$

$ d
 0  ~/tc/config/rates
 1  ~/Projects/projects/taskforest/lib/TaskForest
 2  ~/Documents/Training/qt/ch1
 3  ~
#: 2
$ pwd
/Users/aijaz/Documents/Training/qt/ch1
$ d
 0  ~/Documents/Training/qt/ch1
 1  ~/tc/config/rates
 2  ~/Projects/projects/taskforest/lib/TaskForest
 3  ~
#: q
$

$ p
$ pwd
/Users/aijaz/tc/config/rates
$ d
 0  ~/tc/config/rates
 1  ~/Projects/projects/taskforest/lib/TaskForest
 2  ~
#: q
$

Now let’s have a look at the code that makes this work. You can copy and paste this code directly into your .bashrc file.

# An enhanced 'cd' - push directories
# onto a stack as you navigate to it.
#
# The current directory is at the top
# of the stack.
#
function stack_cd {
    if [ $1 ]; then
        # use the pushd bash command to push the directory
        # to the top of the stack, and enter that directory
        pushd "$1" > /dev/null
    else
        # the normal cd behavior is to enter $HOME if no
        # arguments are specified
        pushd $HOME > /dev/null
    fi
}
# the cd command is now an alias to the stack_cd function
#
alias cd=stack_cd

# Swap the top two directories on the stack
#
function swap {
    pushd > /dev/null
}
# s is an alias to the swap function
alias s=swap

# Pop the top (current) directory off the stack
# and move to the next directory
#
function pop_stack {
    popd > /dev/null
}
alias p=pop_stack

# Display the stack of directories and prompt
# the user for an entry.
#
# If the user enters 'p', pop the stack.
# If the user enters a number, move that
# directory to the top of the stack
# If the user enters 'q', don't do anything.
#
function display_stack
{
    dirs -v
    echo -n "#: "
    read dir
    if [[ $dir = 'p' ]]; then
        pushd > /dev/null
    elif [[ $dir != 'q' ]]; then
        d=$(dirs -l +$dir);
        popd +$dir > /dev/null
        pushd "$d" > /dev/null
    fi
}
alias d=display_stack