Raspberry Pi_Eng_10.9.1 “awk” Command


Published Book on Amazon


All of IOT Starting with the Latest Raspberry Pi from Beginner to Advanced – Volume 1
All of IOT Starting with the Latest Raspberry Pi from Beginner to Advanced – Volume 2


출판된 한글판 도서


최신 라즈베리파이(Raspberry Pi)로 시작하는 사물인터넷(IOT)의 모든 것 – 초보에서 고급까지 (상)
최신 라즈베리파이(Raspberry Pi)로 시작하는 사물인터넷(IOT)의 모든 것 – 초보에서 고급까지 (하)


Original Book Contents


10.9   Commands Related to Data Manipulation

 

10.9.1  "awk" Command

 

The "awk" command reads the specified file, finds the line that has a value matching the specified pattern, and performs the specified operation if the pattern matches. Here, the operation means a field manipulation in a line or an arithmetic operation using a field value. The name "awk" comes from the names of three people who developed it, Aho, Weinberger, and Kernighan.

Awk is a programming language with features such as Shell programming, BC, and the C programming language. It works perfectly with BC, and field variables with names like Shell arguments $1, $2, and $3 can be used on each input line. It also has a printing and control operator similar to the C language.

 

[Command Format]

awk   program    <directory/file>

 

[Command Overview]

   In the specified file, this finds a line with a value matching the specified pattern, and if there is a line matching the pattern, performs the specified operation.

   User privilege          -- Normal user.

 

[Detail Description]

 

 

   How to read "awk" input line

 

When reading data from a standard input or file, each line is divided into several fields based on whitespace. In this command, you can use a variable that represents a field. $1 represents the first field and $2 represents the second field. $0 represents the entire line.

If you want to use a different delimiter instead of a space character, you can use it as a field delimiter by specifying an arbitrary character with "-F" (field) option. For example, to use ":" (colon) as a delimiter, type as follows:

    awk  -F:   program    data-files

 

 

   Program

 

The program expresses the operations on lines and fields that are read from the command. A "program" consists of one or more program lines. Each program line is defined as a pair of pattern and action, and the overall format is as follows:

    pattern {action}

    pattern {action}

 

Consider the example below. Using this program, you can search for lines containing the string "rotate" and print them. This has the same effect as using "grep rotate filename" command:

    /rotate/ {print}   à pattern is "rotate", and action is "print".

 

When defining a pattern, a simple string pattern is defined by enclosing it with "/" before/after the string. You can also use the following unusual patterns:

    /fish/                 line containing the string "fish"

    $1 ~ /fish/            line where the first field contains the string "fish"

    $1 !~ /fish/           line where the first field does not contain the string "fish"

 

In the above, "~" means that the pattern on the right is included in the field on the left. The combination "! ~" means that the pattern on the right is not included in the left field.

The action part is enclosed in curly braces to distinguish it from the pattern. If the action part is omitted, the line is printed. A commonly used action is "print", which prints its arguments to standard output. When several actions are included in { }, separate them with semicolon:

    {print $2}              Print the second field

    {print $4,$2}           Print them in the order of the fourth field and second field

    {print $2,$2+$4}       print the second field, the sum of the second and fourth field

    {s=$2+$4; print s}     Sum the second and fourth fields, and print the sum

 

 

   Examples of various action

 

In the following, we will show examples of various actions. Basically, we will explain using {print $2, $1}. The files generated below are for testing purposes.

 

pi@raspberrypi ~ $ vi  in.file

hello goodbye again

111 222

thirty forty

 

The following example output the two input fields in reverses order. In this example, the arguments of "print" are separated by comma, and it cause to inserts a blank which is the current field separator between the output data. If the comma is omitted, $1 and $2 are printed consecutively.

 

pi@raspberrypi ~ $ vi awk.prog1

{print $2, $1}

pi@raspberrypi ~ $ awk -f  awk.prog1  in.file

goodbye hello

222 111

forty thirty

 

In the following example, for the program of the "awk" command, only one line matches the "/hello/" pattern in the input file, so the specified action "print" will be executed only for the line.

 

pi@raspberrypi ~ $ vi awk.prog2

/hello/ {print $2, $1}

pi@raspberrypi ~ $ awk -f awk.prog2  in.file

goodbye hello

 

In the following example, the parameters added in the second "print" command are enclosed in quotation marks, and the string is added to the existing output and displayed.

 

pi@raspberrypi ~ $ vi awk.prog3

/hello/ {print $2, $1}

/thirty/ {print $1, $2, "and more"}

pi@raspberrypi ~ $ awk -f awk.prog3  in.file

goodbye hello   

thirty forty and more

 

In the following example, the "||" operator executes the specified operation if only one of the two regular expressions matches, and the "&&" operator executes the specified operation when all two regular expressions match. The "!" operator performs the specified operation if the regular expression does not match, and is expressed before the regular expression.

 

pi@raspberrypi ~ $ vi awk.prog4

/hello/||/111/  {print "hit", $1, $2}

pi@raspberrypi ~ $ awk -f awk.prog4  in.file

hit hello goodbye          

  hit 111 222

 

In the following example, only the first input line matching the pattern "/hello/" is the target of the operation.

 

pi@raspberrypi ~ $ vi awk.prog6

/hello/ {

           print $2

           print "another"

           print $1

}

pi@raspberrypi ~ $ awk -f awk.prog6  in.file

goodbye

  another

  hello

 

 

   Arithmetic operation in action of "awk" command

 

The difference between arithmetic operations in "awk" and arithmetic operations in bc is that "awk" can use pattern function to select some of the lines in the input file. For example, the internal function of "awk", "length" returns the length of an input field treated as a string, while a numeric variable can be assigned the value of a field treated as a number, but a string that can not be converted to a number has a value of "0 ".

In the following example, the string "goodbye" or "thirty" can not be converted to a number, so "0" is assigned. However, the string "222" was correctly converted to a number. You can also assign values to variables in the same format as "s=0".

 

pi@raspberrypi ~ $ vi awk.prog7

{

             s += $2

             print $2, "length=" length($2), "s=" s

       }

pi@raspberrypi ~ $ awk -f awk.prog7  in.file

goodbye length=7 s=0   

  222 length=3 s=222

  forth length=5 s=222     

 

 

   "awk" command and variable

 

You can use variables in "awk". Variables do not need to be declared or initialized before use. The "awk" itself initializes variables, and you can store strings or numbers in the variables as needed. The following are correct examples of "awk" variables.
   s,    S,    SS,    S1,    qwerty[42]


Because the type of the variable is automatically converted as shown in the following example, variables can be easily used.

 

pi@raspberrypi ~ $ vi awk.prog8

/hello/ {

             SSS=34

             print "SSS is", SSS

             SSS=hello

             print "SSS is", SSS

           }

pi@raspberrypi ~ $ awk -f awk.prog8  in.file

SSS is 34

    SSS is hello

 

[Main Option]

-f file

program text is read from file instead of from the Shell.  Multiple -f options are allowed.

file: file name that include program instruction

-F(field)

 

-F value

sets the field separator, FS, to value.

If you enter a character, field separator changes into the character

 

 

[Used Example]

The file "sales" has six columns of information. From the beginning, item name, item selling price, and the item sales quantity for each quarter. Here we use the "awk" command to calculate the sum of the item sales quantity and the item sales amount, and add them to the file. To do this, create a file named "addup" as follows.

 

pi@raspberrypi ~ $ vi sales

carts      29.99  45  13  55  22

    corks        0.02  30  20  25  93

    doors     49.99  40  15  20  25

    geese     15.00   2   8   1 128

    nighties   50.00  11  23  15  82

pi@raspberrypi ~ $ vi addup

{total=$3+$4+$5+$6;print $0, total, total*$2}

pi@raspberrypi ~ $ awk -f addup sales

carts      29.99  45  13  55  22         135  4040.64

    corks               0.02  30  20  25  93          93     3.36

    doors     49.99  40  15  20  25         100  4999 

    geese     15.00   2   8   1 128         139  2085 

    nighties  50.00  11  23  15  82         131  6550 

 

The above action consists of two parts separated by ";". The first part adds the sales quantities and stores them in the "total" variable. The second part prints the original line ($0), followed by the sales quantity, followed by the value of total * $2, which means "sales quantity x second column value".