HPR1648 - full show notes

Bash parameter manipulation

I'm a great fan of using the Linux command line and enjoy writing shell scripts using the Bash shell.

In this episode we look at what parameters are in Bash, and how they can be created and manipulated. There are many features in Bash that you can use to do this, but they are not easy to find.

As I was learning my way around Bash it took me a while to find these. Once I had found them I wanted to make a "cheat sheet" I could stick on the wall to remind me how to do things. I am sharing the result of this process with you.

The version of Bash which I used for this episode is 4.3.30(1)-release

What is a parameter?

A Bash parameter, more commonly referred to as a variable, is a named item which holds a value. Parameters are created thus:

username='droog'

There should be no spaces before or after the '=' sign. The parameter called username now contains the string 'droog'.

To use the contents of username the name should be prefixed with a dollar ($) sign as in:

echo "Your username is $username"
-> Your username is droog

The line beginning '->' is what will be generated by the above statement. I will be using this method of signifying output through these notes.

An alternative way of referring to the contents of a parameter is to enclose it with curly brackets (braces) after the dollar sign. This makes the variable name unambiguous when there is a possibility of misinterpretation

Arrays

As well as the simple parameters seen so far, Bash also provides arrays. The simplest form is indexed by integer numbers, starting with zero and is defined either as a bracketed list:

weekdays=(monday tuesday wednesday thursday friday)

or individually as indexed elements:

weekend[0]='saturday'
weekend[1]='sunday'

There is a lot more to arrays in Bash than this, as well as there being associative arrays indexed by strings, but we will leave them for another time.

Referring to arrays is achieved using curly braces and indices:

echo ${weekdays[4]}
-> friday

The entire array can be referenced with '@' or '*' as an index:

echo ${weekdays[@]}
-> monday tuesday wednesday thursday friday

Knowing this much about arrays is necessary to understand the following parameter manipulation expressions.

Manipulating parameters

If you want to change the value of a parameter there are many ways of doing it. A lot of scripts you may encounter might use sed, awk or cut to do this. For example, you might see this:

date='2014-10-27'
month=$(echo $date | cut -f2 -d'-')
echo "The month number is $month"
-> The month number is 10

Here cut was used to do the job of extracting the '10' from the contents of date. However, this is inefficient since it causes a whole new process to be run just to do this simple thing. Bash contains the facilities to do this all by itself. Here's how, using substring expansion:

month=${date:5:2}

Variable month is set to the characters of date from position 5 for 2 characters (the position is zero based by the way).

Parameter manipulation features

I have demonstrated each of these briefly. Included with these notes are two other resources: the relevant text from the bash man page and some examples in diagrammatic form. Both are PDF files generated from LibreOffice.

The man page extract was originally made for my own benefit as a cheat sheet to help to remind me how to use these features. If it benefits you then great. If not then no problem.

The diagram was also meant for me to place on a pin-board over my desk, so it includes colour and seems a little more friendly. If you like it then you're welcome.

I have also included the examples below, with a little more explanation. I hope it helps.

Use default values

Returns the value of the parameter unless it's not defined or is null, in which case the default is returned:

unset name
echo ${name:-Undefined}
-> Undefined
name="Charlie"
echo ${name:-Undefined}
-> Charlie

Assign default values

Set the value of the parameter if it is undefined or null:

unset page
echo ${page:=portrait}
-> portrait
echo $page
-> portrait
page="landscape"
echo ${page:=portrait}
-> landscape
echo $page
-> landscape

Display Error if Null or Unset

Displays an error and causes the enclosing script to exit if the parameter is not set (the error message contains details of where in the script the problem occurred):

echo ${length:?length is unset}
-> ... length: length is unset

Use Alternate Value

If the parameter is null or unset then nothing is substituted.

fish="trout"
echo ${fish:+salmon}
-> salmon
echo $fish
-> trout

Substring Expansion

This one is quite complex. The first value is an offset and the second a length. If the length is omitted then the rest of the string is returned.

A negative offset means to count backwards from the end of the string. Note that the sign must be preceded by a space to avoid being misinterpreted as the default value form.

A negative length is not really a length, but means to return the string between the offset and the position backwards from the end of the string.

Sections of arrays may also be indexed with this expression. The offset is an offset into the elements of the array and the length is a count of elements.

animal="aardvark"
echo ${animal:4}
-> vark
message="No such file"
echo ${message:0:7}
-> No such
echo ${message: -4}
-> file
echo ${message:3:-4}
-> such

colours=(red orange yellow green blue indigo violet)
echo ${colours[@]:1:3}
-> orange yellow green
echo ${colours[@]:5}
-> indigo violet

Names matching prefix

This is for reporting names of variables. We do not show all the names in the examples below.

echo ${!BASH*}
-> BASH BASHOPTS BASHPID ...
export coord_x=42
export coord_y=100
echo ${!coord*}
-> coord_x coord_y

List of array keys

Lists the array indices (more generally keys) in an array.

colours=( red green blue )
echo ${!colours[@]}
-> 0 1 2

Parameter length

Shows the length of a parameter. If used with an array it returns the number of elements.

Note the second example below saves the result of the date command in an array dt. There are 6 fields separated by spaces, so the array element count reflects this.

veg="Broccoli"
echo ${#veg}
-> 8
dt=($(date))
echo ${#dt[@]}
-> 6

Remove matching prefix pattern

This removes characters from the front of a string. The pattern used can contain an asterisk ('*') meaning an arbitrary number of characters. If two hash characters ('#') are used the longest match is removed.

So, in the first examples '*/' with one hash just removes the leading '/', but with two hashes everything up to and including the last '/' is removed. This is equivalent to the built-in basename command.

When applied to an array every element can be trimmed as shown in the second example. Note that here we are saving the result of trimming the leading '_' back into the array.

dir="/home/dave/some/dir"
echo ${dir#/home/dave/}
-> some/dir
echo ${dir#*/}
-> home/dave/some/dir
echo ${dir##*/}
-> dir

colours=(_red _green _blue)
colours=(${colours[@]#_})
echo ${colours[@]}
-> red green blue

Remove matching suffix pattern

This feature is similar to the previous one and removes characters from the end of a string. The pattern used to determine what to remove is the same as before, and the use of double '%' characters makes the deletion affect the maximum number of characters.

Note that using '/*' in the examples has an effect similar to the dirname command.

A common use is shown in the second example where the extension of a filename is deleted and replaced.

dir="/home/dave/some/dir"
echo ${dir%/some/dir}
-> /home/dave
echo ${dir%/*}
-> /home/dave/some
echo ${dir%%/some/*}
-> /home/dave

filename='great_view.jpg'
filename="${filename%.jpg}.png"
echo $filename
-> great_view.png

Pattern substitution

This feature permits quite sophisticated changes to be made to a string or an array. The first part after the first '/' is the pattern to match, and can contain '*' as described before. See the Bash Hackers site for a very good explanation of this. The second string is what is to replace the target.

If two '/' characters follow the parameter name then all matches that are found are replaced.

If the pattern string begins with a '#' then it must match at the start of the parameter, however if the pattern string begins with a '%' then it must match at the end of the parameter.

msg='An ant is an ant'
echo ${msg/ant/insect}
-> An insect is an ant
echo ${msg/%ant/insect}
-> An ant is an insect
echo ${msg//ant/insect}
-> An insect is an insect

colours=(red green blue)
echo ${colours[@]/green/yellow}
-> red yellow blue
echo ${colours[@]/#/_}
-> _red _green _blue

Case modification

Finally, this feature changes the case of letters. The '^' symbol makes matching letters into uppercase, and ',' converts to lowercase. A single '^' or ',' after the parameter name makes one change, whereas doubling this symbol changes every matching character.

Note the use of a pattern enclosed in square brackets matches the enclosed letters. Adding a '^' to the start of this list inverts the matching effect as seen below.

msg='the quick brown fox'
echo ${msg^}
-> The quick brown fox
echo ${msg^^}
-> THE QUICK BROWN FOX
echo ${msg^^o}
-> the quick brOwn fOx
echo ${msg^^[tqbf]}
-> The Quick Brown Fox
echo ${msg^^[^tqbf]}
-> tHE qUICK bROWN fOX