Thanks for all your error reports, I didn't forget it. I'll cleanup my guide soon. Thanks again!

Exercise 7. Bash: redirection, stdin, stdout, stderr, <, >, >>, |, tee, pv

In Linux, everything is just a file. This means that, for console programs:

  1. Keyboard is represented as a file, from which Bash reads your input.
  2. Display is represented as a file, to which Bash writes its output.

Let us say you have a program which counts lines in a file. You can call it by typing wc -l. Try this now. Nothing happened, right? It just hangs there. Wrong, it is waiting for your input. This is an outline of what it does:

line_counter = 0
while end of file is not reached
    read a line
    add 1 to line_counter
print line_counter

So wc currently reads lines from /dev/tty, which is in current context your keyboard. A line is given to wc each time you hit Enter. Type any several lines, and then hit CTRL+D, which will produce EOF character for wc to understand that end of file is reached. Now it will tell you how much lines you entered.

But what if you want to count lines in existing file? Well, where is redirection for that! To feed an existing file on its input, type this: wc -l < .bash_history. You see, it works! And it really is that simple! redirection is a mechanism which allows you tell a program to redirect its input and/or output from keyboard and display to another file. To do this, you can use these special commands then starting a program:

  1. < — replace standard input (for example keyboard) with a file.
  2. > — replace standard output (for example display) with a file. Warning! This command overwrites contents of file you specify, so be careful.
  3. >> — same as above, but instead of overwriting a file it writes to the end of it, preserving already existing information in this file. Be careful not to confuse the two.
  4. | — Take an output from one program and connect it to the other. This will be elaborated on in next exercise.

Now you will learn how to redirect input and output of programs to files or another programs.

Do this

 1: sudo aptitude install pv
 2: read foo < /dev/tty
 3: Hello World!
 4: echo $foo > foo.out
 5: cat foo.out
 6: echo $foo >> foo.out
 7: cat foo.out
 8: echo > foo.out
 9: cat foo.out
10: ls -al | grep foo
11: ls -al | tee ls.out
12: dd if=/dev/zero of=~/test.img bs=1M count=10
13: pv ~/test.img | dd if=/dev/stdin of=/dev/null bs=1
14: rm -v foo.out
15: rm -v test.img

What you should see

user1@vm1:~$ sudo aptitude install pv
The following NEW packages will be installed:
  pv
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/28.9 kB of archives. After unpacking 143 kB will be used.
Selecting previously deselected package pv.
(Reading database ... 39657 files and directories currently installed.)
Unpacking pv (from .../archives/pv_1.1.4-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up pv (1.1.4-1) ...
 
user1@vm1:~$ read foo < /dev/tty
Hello World!
user1@vm1:~$ echo $foo > foo.out
user1@vm1:~$ cat foo.out
Hello World!
user1@vm1:~$ echo $foo >> foo.out
user1@vm1:~$ cat foo.out
Hello World!
Hello World!
user1@vm1:~$ echo > foo.out
user1@vm1:~$ cat foo.out
 
user1@vm1:~$ ls -al | grep foo
-rw-r--r-- 1 user1 user1    1 Jun 15 20:03 foo.out
user1@vm1:~$ ls -al | tee ls.out
total 44
drwxr-xr-x 2 user1 user1 4096 Jun 15 20:01 .
drwxr-xr-x 3 root  root  4096 Jun  6 21:49 ..
-rw------- 1 user1 user1 4865 Jun 15 19:34 .bash_history
-rw-r--r-- 1 user1 user1  220 Jun  6 21:48 .bash_logout
-rw-r--r-- 1 user1 user1 3184 Jun 14 12:24 .bashrc
-rw-r--r-- 1 user1 user1    1 Jun 15 20:03 foo.out
-rw------- 1 user1 user1   50 Jun 15 18:41 .lesshst
-rw-r--r-- 1 user1 user1    0 Jun 15 20:03 ls.out
-rw-r--r-- 1 user1 user1  697 Jun  7 12:25 .profile
-rw-r--r-- 1 user1 user1  741 Jun  7 12:19 .profile.bak
-rw-r--r-- 1 user1 user1  741 Jun  7 13:12 .profile.bak1
-rw-r--r-- 1 user1 user1    0 Jun 15 20:02 test.img
user1@vm1:~$ dd if=/dev/zero of=~/test.img bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0130061 s, 806 MB/s
user1@vm1:~$ pv ~/test.img | dd if=/dev/stdin of=/dev/null bs=1
  10MB 0:00:03 [3.24MB/s] [=================================================================================>] 100%
10485760+0 records in
10485760+0 records out
10485760 bytes (10 MB) copied, 3.10446 s, 3.4 MB/s
user1@vm1:~$ rm -v foo.out
removed `foo.out'
user1@vm1:~$ rm -v test.img
removed `test.img'
user1@vm1:~$

Explanation

  1. Install pv (pipe viewer) program on your system which you will need later on.
  2. Reads your input to a variable foo. This is possible because display and keyboard are actually a teletype for the system. Yes, that teletype! Read more about tty online.
  3. This is what you type.
  4. Redirects contents of the foo variable to the foo.out file, creating the file in the process or overwriting an existing file without warning removing all there was!!
  5. Prints out foo.out contents.
  6. Redirects contents of the foo variable to the foo.out file, creating the file in the process or appending to an existing file. This is safe, but don not confuse the two, or great sorrow will fall upon you.
  7. Prints out foo.out contents again.
  8. Redirects nothing to the foo.out, emptying file in the process.
  9. Shows that the file is, indeed, empty.
  10. Lists your directory and pipes the output to grep. This works by taking all ls -al output and feeding it into grep. Again, this is called piping.
  11. Lists your directory to the screen and to the ls.out. Very useful!
  12. Creates a zeroed file with size of 10 Megabytes. Don't bother with how this works for now.
  13. Reads this file to /dev/null, which is an ultimate garbage can in your system, a void. Everything which is written into it goes nowhere. Notice how pv shows you a progress of reading the file, if you try to read it with another command instead you won't know how long it will take to complete.
  14. Removes foo.out. Always remember to clean up after yourself.
  15. Removes test.img.

Extra credit

  1. Read about piping and redirection on stackoverflow, again on stackoverflow and on Greg's Wiki which is very useful resource indeed, remember it.
  2. Open man page for bash, scroll down to REDIRECTION section and read it.
  3. Read descriptions from man pv and man tee.

Discussion

Navigation

Learn Linux The Hard Way