Introduction to the core utilities, part 2.

It should come as no suprise that there are dozens of small utilities that are included with every Linux distribution.  These are the core utiliites, and this is a brief introduction to the displaying and formating tools.  Part 2 covers some utilities to output parts of files:

head

head displays the first n lines (or n bytes) of a file (or standard input).  By default head displays the first 10 lines of a file, but that is easily changed:

Display the first 10 lines of /etc/passwd

    ~/projects/pt2> head /etc/passwd

    root:x:0:0:root:/root:/bin/bash

    daemon:x:1:1:daemon:/usr/sbin:/bin/sh

    bin:x:2:2:bin:/bin:/bin/sh

    sys:x:3:3:sys:/dev:/bin/sh

    sync:x:4:65534:sync:/bin:/bin/sync

    games:x:5:60:games:/usr/games:/bin/sh

    man:x:6:12:man:/var/cache/man:/bin/sh

    lp:x:7:7:lp:/var/spool/lpd:/bin/sh

    mail:x:8:8:mail:/var/mail:/bin/sh

    news:x:9:9:news:/var/spool/news:/bin/sh

The first 10 lines of /etc/passwd  passed to a script to list the usernames and their shells:

    ~/projects/pt2> head /etc/passwd | cut -d: -f1,7

    root:/bin/bash

    daemon:/bin/sh

    bin:/bin/sh

    sys:/bin/sh

    sync:/bin/sync

    games:/bin/sh

    man:/bin/sh

    lp:/bin/sh

    mail:/bin/sh

    news:/bin/sh

Display the first 5 lines of the output of dmesg

    ~/projects/pt2> dmesg | head -n 5

    [    0.000000] Initializing cgroup subsys cpuset

    [    0.000000] Initializing cgroup subsys cpu

    [    0.000000] Linux version 2.6.32-5-686 (Debian 2.6.32-39) (dannf@debian.org) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Thu Nov 3 04:23:54 UTC 2011

    [    0.000000] KERNEL supported cpus:

    [    0.000000]   Intel GenuineIntel

Display the first 2 GigaBytes of the 1st partition of the second hard drive, then passed to strings to get a listing of all printable characters

    ~/projects/pt2> head -c 2G /dev/sdb1 | strings

    [output omitted for brevity]

tail

tail displays the last n lines (or n bytes) of a file (or standard input)

Display the last 10 lines of a syslog, and then each line that comes into said file (usefull watching log files, for instance)

    ~/projects/pt2# tail  /var/log/syslog

    Dec  2 09:16:27 valhalla dhclient: DHCPACK from 192.168.200.1

    Dec  2 09:16:27 valhalla dhclient: bound to 192.168.200.106 -- renewal in 42062 seconds.

    Dec  2 20:57:29 valhalla dhclient: DHCPREQUEST on eth0 to 192.168.200.1 port 67

    Dec  2 20:57:29 valhalla dhclient: DHCPACK from 192.168.200.1

    Dec  2 20:57:29 valhalla dhclient: bound to 192.168.200.106 -- renewal in 39735 seconds.

    Dec  3 07:30:01 valhalla anacron[31196]: Anacron 2.3 started on 2011-12-03

    Dec  3 07:30:01 valhalla anacron[31196]: Will run job `cron.daily' in 5 min.

    Dec  3 07:30:01 valhalla anacron[31196]: Jobs will be executed sequentially

    Dec  3 07:35:01 valhalla anacron[31196]: Job `cron.daily' started

    Dec  3 07:35:01 valhalla anacron[31202]: Updated timestamp for job `cron.daily' to 2011-12-03

Display lines 450-500 (or the last 50 lines of a file if it doesn't have at least 500 lines in it)

     ~/projects/pt2# head -n 500 /var/log/messages | tail -n 50

     [output omitted for brevity]

GNU tail can also keep track of multiple files, output the last n bytes of a file, and can be made to auto terminate nate.     Monitor both /var/log/messages and /var/log/syslog

    ~/projects/pt2> tail -f /var/log/{syslog,messages}

    ==> /var/log/syslog <==

    Dec  9 15:26:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 15:36:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 15:46:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 15:56:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:06:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:16:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:17:01 valhalla /USR/SBIN/CRON[18156]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

    Dec  9 16:26:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:36:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:46:05 valhalla mpt-statusd: detected non-optimal RAID status

    ==> /var/log/messages <==

    Dec  9 15:16:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 15:26:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 15:36:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 15:46:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 15:56:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:06:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:16:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:26:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:36:05 valhalla mpt-statusd: detected non-optimal RAID status

    Dec  9 16:46:05 valhalla mpt-statusd: detected non-optimal RAID status

split

split takes a file and splits it into several smaller files. For example, say you have an email limit, where each file can only be 1 mb in size, and you need to email a 5 mb jpg.  You could split the file thusly:

     ~/projects/pt2> split -b 750000 Blinds.jpg Blinds.jpg.part.

     ~/projects/pt2> ls -gG Blinds*

     -rw-r--r-- 1 1157513 Dec  9 17:25 Blinds.jpg

     -rw-r--r-- 1  750000 Dec  9 17:26 Blinds.jpg.part.aa

     -rw-r--r-- 1  407513 Dec  9 17:26 Blinds.jpg.part.ab

And then email the parts one by one.  Why 750,000 bytes?  MIME encoding the file will increase each part to about 133% of its starting size, and 750,000 * 133 = 997,500.  This leaves us some room for the email message and header.

To join the files back into 1 piece, simply cat them together, like so:

     ~/projects/pt2> cat Blinds.jpg.part.* > Blinds.jpg

If the system that you are joining them together on is Windows, you can join together files splitted with a comm and like this:

     C:\> copy /b Blinds.jpg.part.aa+Blinds.jpg.part.ab Blinds.jpg

split can also run a filter on the parts as it splits them.

Split the weather.log file into parts, compressing them as they go.

     ~/projects/pt2> split --filter 'gzip -9c > $FILE.gz' weather.log weather-part-

Next week we'll continue on with our tour of the core utilities, working with file hashes and various ways to sort files.

Introduction to the GNU core utilities

It should come as no suprise that there are dozens of small utilities that are included with every Linux distribution.  These are the core utilities, and hopefully you'll find these examples somewhat useful:


Utilities used to display files

cat
cat display a file.  By itself, kind of boring, but there are a couple of options that could come in handy:
cat -b   # Number all nonblank line
cat -n   # Number all lines (though nl is more specialized)
cat -s   # Squeeze 2 or more blank lines into 1
cat -E   # Display's a $ at the end of the line
cat -T   # Display's a ^I for each tab character

~/projects/articles/coreutils> cat -nET helloworld.c
1  #include <stdio.h>$
2  $
3  int main(void) {$
4     printf("Hello World!\n");$
5  ^Ireturn 0;         $
6  }$

tac
tac displays a file backwords, bottom up.

~/projects/articles/coreutils> tac helloworld.c | cat -nE
1  }$
2          return 0;         $
3     printf("Hello World!\n");$
4  int main(void) {$
5  $
6  #include <stdio.h>$


nl
nl numbers lines in a file.  By default it only numbers non empty lines, but it still displays them:


~/projects/articles/coreutils> nl -nrz hello.c
000001  #include <stdio.h>

000002  int main(void) {
000003     printf("Hello World!\n");
000004          return 0;
000005  }

add the -ba option and nl will number all lines in the file, and a -w3 to set the

~/projects/articles/coreutils> nl -nrz -w3 -ba hello.c
001     #include <stdio.h>
002
003     int main(void) {
004        printf("Hello World!\n");
005             return 0;
006     }


od
od displays a file in standard hex dump format.  Actually, hex is just one of the options, the default is octal, hence the name  (o)ctal (d)ump.


~/projects/articles/coreutils> od -x hello.c
0000000 6923 636e 756c 6564 3c20 7473 6964 2e6f
0000020 3e68 0a20 690a 746e 6d20 6961 286e 6f76
0000040 6469 2029 0a7b 2020 7020 6972 746e 2866
0000060 4822 6c65 6f6c 5720 726f 646c 5c21 226e
0000100 3b29 090a 6572 7574 6e72 3020 203b 2020
0000120 2020 2020 2020 7d0a 000a
0000131

You can also intersperse the ascii equivalent:


~/projects/articles/coreutils> od -ax hello.c
0000000   #   i   n   c   l   u   d   e  sp   <   s   t   d   i   o   .
6923    636e    756c    6564    3c20    7473    6964    2e6f
0000020   h   >  sp  nl  nl   i   n   t  sp   m   a   i   n   (   v   o
3e68    0a20    690a    746e    6d20    6961    286e    6f76
0000040   i   d   )  sp   {  nl  sp  sp  sp   p   r   i   n   t   f   (
6469    2029    0a7b    2020    7020    6972    746e    2866
0000060   "   H   e   l   l   o  sp   W   o   r   l   d   !   \   n   "
4822    6c65    6f6c    5720    726f    646c    5c21    226e
0000100   )   ;  nl  ht   r   e   t   u   r   n  sp   0   ;  sp  sp  sp
3b29    090a    6572    7574    6e72    3020    203b    2020
0000120  sp  sp  sp  sp  sp  sp  nl   }  nl
2020    2020    2020    7d0a    000a
0000131

Although if you want that then you might be more interested in the traditional hexdump:
~/projects/articles/coreutils> hexdump -C hello.c
00000000  23 69 6e 63 6c 75 64 65  20 3c 73 74 64 69 6f 2e  |#include <stdio.|
00000010  68 3e 20 0a 0a 69 6e 74  20 6d 61 69 6e 28 76 6f  |h> ..int main(vo|
00000020  69 64 29 20 7b 0a 20 20  20 70 72 69 6e 74 66 28  |id) {.   printf(|
00000030  22 48 65 6c 6c 6f 20 57  6f 72 6c 64 21 5c 6e 22  |"Hello World!\n"|
00000040  29 3b 0a 09 72 65 74 75  72 6e 20 30 3b 20 20 20  |);..return 0;   |
00000050  20 20 20 20 20 20 0a 7d  0a                       |      .}.|
00000059



base64
base64 takes a binary file and transcodes it into printable ASCII characters.  This does inflate the size of a file to about 133% of its original size.  Base64 files are used in mediums that aren't designed to handle binary streams (such as nntp, or if you really are desperate to transer a file, copying a file through a telnet session) uuencode is another type of encoding and pgp uses ASCII armour after it encrypts messages for emailing.

Here is a base64 encoded message:
~/projects/articles/coreutils> cat hello.64
I2luY2x1ZGUgPHN0ZGlvLmg+IAoKaW50IG1haW4odm9pZCkgewogICBwcmludGYoIkhlbGxvIFdv
cmxkIVxuIik7CglyZXR1cm4gMDsgICAgICAgICAKfQo=

and the same message decoded.
~/projects/articles/coreutils> base64 -d < hello.64
#include <stdio.h>


int main(void) {
printf("Hello World!\n");
return 0;
}

And the same file uuencoded:
~/projects/articles/coreutils> uuencode hello.c < hello.c
begin 644 hello.c
M(VEN8VQU9&4@/'-T9&EO+F@^(`H*:6YT(&UA:6XH=F]I9"D@>PH@("!P<FEN
L=&8H(DAE;&QO(%=O<FQD(5QN(BD["@ER971U<FX@,#L@("`@("`@("`*?0H`
`
end


Utilities for formatting files

fmt
fmt takes texts and formates it so that each line takes no more then a certain width.  For its purpose, fmt assumes that all the lines in a paragraph are meant to stay together, so a short line will be merged with a longer line

For example:
(Example of a short line followed by a really long line):
~/projects/articles/coreutils> cat grimm
A certain king had a beautiful garden, and in the garden stood a tree
which bore golden apples. These apples were always counted, and about
the time when they began to grow ripe it was found that every night one
of them was gone. The king became very angry at this, and ordered the
gardener to keep watch all night under the tree. The gardener set his
eldest son to watch; but about twelve o'clock he fell asleep, and in
the morning another of the apples was missing. Then the second son was
ordered to watch; and at midnight he too fell asleep, and in the morning
another apple was gone. Then the third son offered to keep watch; but
the gardener at first would not let him, for fear some harm should come
to him: however, at last he consented, and the young man laid himself
under the tree to watch. As the clock struck twelve he heard a rustling
noise in the air, and a bird came flying that was of pure gold; and as
it was snapping at one of the apples with its beak, the gardener's son
jumped up and shot an arrow at it. But the arrow did the bird no harm;
only it dropped a golden feather from its tail, and then flew away.
The golden feather was brought to the king in the morning, and all the
council was called together. Everyone agreed that it was worth more than
all the wealth of the kingdom: but the king said, 'One feather is of no
use to me, I must have the whole bird.'


Running it through fmt:
~/projects/articles/coreutils> fmt -40 grimm
A certain king had a beautiful garden,
and in the garden stood a tree which
bore golden apples. These apples were
always counted, and about the time
when they began to grow ripe it was
found that every night one of them was
gone. The king became very angry at
this, and ordered the gardener to keep
watch all night under the tree. The
gardener set his eldest son to watch;
but about twelve o'clock he fell
asleep, and in the morning another
of the apples was missing. Then the
second son was ordered to watch; and at
midnight he too fell asleep, and in the
morning another apple was gone. Then
the third son offered to keep watch;
but the gardener at first would not let
him, for fear some harm should come to
him: however, at last he consented,
and the young man laid himself under
the tree to watch. As the clock struck
twelve he heard a rustling noise in the
air, and a bird came flying that was
of pure gold; and as it was snapping
at one of the apples with its beak,
the gardener's son jumped up and shot
an arrow at it. But the arrow did the
bird no harm; only it dropped a golden
feather from its tail, and then flew
away.  The golden feather was brought
to the king in the morning, and all the
council was called together. Everyone
agreed that it was worth more than all
the wealth of the kingdom: but the king
said, 'One feather is of no use to me,
I must have the whole bird.'


pr
pr takes standard input and formats it for printing by puting a header on the text.  By default pr assumes that a page is 66 rows and 72 columns:

~/projects/articles/coreutils> pr -f grimm




2011-12-04 15:40                      grimm                       Page 1




A certain king had a beautiful garden, and in the garden stood a tree
which bore golden apples. These apples were always counted, and about
the time when they began to grow ripe it was found that every night one
of them was gone. The king became very angry at this, and ordered the
gardener to keep watch all night under the tree. The gardener set his
eldest son to watch; but about twelve o'clock he fell asleep, and in
the morning another of the apples was missing. Then the second son was
ordered to watch; and at midnight he too fell asleep, and in the morning
another apple was gone. Then the third son offered to keep watch; but
the gardener at first would not let him, for fear some harm should come
to him: however, at last he consented, and the young man laid himself
under the tree to watch. As the clock struck twelve he heard a rustling
noise in the air, and a bird came flying that was of pure gold; and as
it was snapping at one of the apples with its beak, the gardener's son
jumped up and shot an arrow at it. But the arrow did the bird no harm;
only it dropped a golden feather from its tail, and then flew away.
The golden feather was brought to the king in the morning, and all the
council was called together. Everyone agreed that it was worth more than
all the wealth of the kingdom: but the king said, 'One feather is of no
use to me, I must have the whole bird.'
^L

If you want to print the text as if it were a newspaper, you need to make sure that each line is less the width of the columns (ie: 72/2==36 for 2 columns by default),  one way to do that is to use fmt on the text:

~/projects/articles/coreutils> fmt -35 grimm | pr -2f |more




2011-12-04 15:48                                                  Page 1




A certain king had a beautiful      should come to him: however,
garden, and in the garden           at last he consented, and the
stood a tree which bore golden      young man laid himself under
apples. These apples were always    the tree to watch. As the clock
counted, and about the time when    struck twelve he heard a rustling
they began to grow ripe it was      noise in the air, and a bird came
found that every night one of them  flying that was of pure gold;
was gone. The king became very      and as it was snapping at one
angry at this, and ordered the      of the apples with its beak,
gardener to keep watch all night    the gardener's son jumped up
under the tree. The gardener set    and shot an arrow at it. But the
his eldest son to watch; but about  arrow did the bird no harm; only
twelve o'clock he fell asleep,      it dropped a golden feather from
and in the morning another of       its tail, and then flew away.
the apples was missing. Then        The golden feather was brought
the second son was ordered to       to the king in the morning,
watch; and at midnight he too       and all the council was called
fell asleep, and in the morning     together. Everyone agreed that it
another apple was gone. Then the    was worth more than all the wealth
third son offered to keep watch;    of the kingdom: but the king said,
but the gardener at first would     'One feather is of no use to me,
not let him, for fear some harm     I must have the whole bird.'
^L


fold 
fold breaks long lines into multiple lines (each of 80 characters), without regard to formating.  There is an option to break on word boundaries, if desired.

~/projects/articles/coreutils> fold -35 -s  grimm  | head
A certain king had a beautiful
garden, and in the garden stood a
tree
which bore golden apples. These
apples were always counted, and
about
the time when they began to grow
ripe it was found that every night
one
of them was gone. The king became


That does it for now.  In the future I'll give examples for the file output and sorting utilities.