Bash (Linux Shell)

Linux shell scripting language

Cut #1 - 4 (easy cuts)

Linux Cut Command Help and Examples

# OVERVIEW
## 1. Cut third field of each line - Output : three \n gamma
cut -f 3 data.txt
## 2. Cut third field and every field after it, omit first 2 fields
cut -f 3- data.txt
>> three    four    five
   gamma   delta   epsilon
## 3. 3rd char of every line in a file
cut -c 3 file.txt
## 4. output first 3 char of every line 
cut -c 1-3 file.txt
## 5. output first 3 char and omit the rest / last 3 char 
cut -c -3 file.txt / cut -c 3- file.txt


## 6. Output the first field of the file /etc/passwd, where fields are
### delimited by a colon (':'). The first field of /etc/passwd is the username,      
### so this command outputs every username in the passwd file.
cut -d ':' -f 1 /etc/passwd

## 7. Otput 1st and 6th fields, delimited by colon, of any entry in the /etc/passwd file
### which specifies /bin/bash as the login shell. This command outputs the username 
### and home directory of any user whose login shell is /bin/bash.
grep '/bin/bash' /etc/passwd  cut -d ':'-f 1,6

## 8. Counting bytes
cut -b 3-12 data.txt

## 9. To add delimiter to output (adding space in this case)
cut -f 1,3 -d ':' --output-delimiter=' ' /etc/passwd
##10. To add tab delimiter for output (tab is special case, cause it is unprintable)   
cut -f 1,3 -d ':' --output-delimiter=$'\t' /etc/passwd #use $ to protect from shell

/etc/passwd File

For instance, the file /etc/passwd contains information about each user on the system, one user per line, and each information field is delimited by a colon (":"). For example, the line of /etc/passwd for the root user may look like this:

root:x:0:0:root:/root:/bin/bash

These fields contain the following information, in the following order, separated by a colon character:

1. Username

2. Password (shown as x if encrypted)

3. User ID number or UID

4. Group ID number or GID

5. Comment field (used by the finger command)

6. Home Directory

7. Shell

It is often used for parsing data from log files, csv files, and similar. Given a firewall's log file, for example, cut and sort -u can be used to create a sorted list of unique ip addresses. Add in grep to get a strong tool for carving out data. The following lines show how to extract data from a line of text.

Example below:

The -d flag sets the delimiter, space in this case, and the -f flag shows which column to return, . The column count starts at .

In the next command, output from the first command is piped to a second command where the delimiter is a period and the column is . Finally, cut is used to extract the first character from the results of the second command.

echo '0000 192.168.1.100 192.168.100.1' |cut -d ' ' -f 2 
192.168.1.100
echo '0000 192.168.1.100 192.168.100.1' |cut -d ' ' -f 2 |cut -d '.' -f 4
100
echo '0000 192.168.1.100 192.168.100.1' |cut -d ' ' -f 2 |cut -d '.' -f 4|cut -c 1
1

The above is for command line inputs.

HackerRank Task : Read N lines of input, then output the 3rd character of each input. (etc. Input = character, output = a)

#!/bin/bash
while read line # while there is input, line is th variable here, 
#read is the function to read input
do 
echo $line | cut -c 3
done 

# to output characters 2nd and 7th
while 
read line
do 
echo $line | cut -c 2,7
done

Cut #5 - (tabs)

1. No quote vs Double Quotes

When double quotes are used white space (tab space is a form of white space) is preserved. When double quotes aren't used tabspace isn't preserved. cut by default uses tabspace as a delimiter to extract the fields. If no tabspaces are present (and if delimiter option of cut hasn't been modified) then cut wont work.

2. IFS (Internal Field Separator)

It is used by the shell to determine how to do word splitting, i. e. how to recognize word boundaries. The default value for IFS consists of whitespace characters (to be precise: space, tab and newline). Each character can be a word boundary.

#!/bin/bash
### TASKS ANSWER
while read line
do
echo $line | cut -f1-3  ## METHOD 1. WILL NOT WORK
## VERSUS
echo "$line" | cut -f1-3 ## METHOD 2. WORKS
done

## USING IFS
IFS="" # NO SPACES BETWEEN THE IFS AND = sign
while read line; do
  echo -e $line | cut -f -3
done

Head & Tails

Used to display first few lines of a file or last few lines of file

head [filename]
head -n 11 [filename]  -> First 11 lines  
head -c 20 [filename]  -> First 20 characters  
# if read from STDIN, ignore filename
head -n 20
tail -n 20 # last twenty lines

## To get middle of text file (to get lines 12-22)
head -n 22 | tail -n 11 # get first 22, then get last 11 of these 22 lines

tr (translate)

Translate offers various ways that the tr command may be used for transforming or translating characters or character classes. It can also be used to delete, complement, remove or squeeze characters

-c : complements the set of characters in string.i.e., operations apply to characters not in the given set

-d : delete characters in the first set from the output.

-s : replaces repeated characters listed in the set1 with single occurrence

-t : truncates set1

For example: 'e' being transformed to 'E'

$ echo "Hello" | tr "e" "E"
HEllo

Spaces being transformed to hyphens

$ echo "Hello how are you" | tr " " '-'
Hello-how-are-you

Digits (numerals) being deleted

$ echo "Hello how are you 1234" | tr -d [0-9]
Hello how are you 

Deleting lowercases a-z

tr -d [a-z] # flag of d for delete

Replace multiple spaces with single space

tr -s " "