Bash Tips - 18 (HPR Show 2729)

Dave Morriss


Table of Contents

Arrays in Bash

This is the third of a small group of shows on the subject of arrays in Bash. It is also the eighteenth show in the Bash Tips sub-series.

In the last show we looked at ways of accessing elements with negative indices and how to concatenate arrays. We then launched into parameter expansion in the context of arrays.

There are a few more parameter expansion operations to look at in this episode, then in the next episode we will look in more depth at the declare built in command and at some of the commands that assist with loading data into arrays.

More parameter expansion operations and arrays

String replacement

This expansion performs a single replacement within a parameter string or repeats the replacement throughout the entire string. It can also perform the same type of operations on array elements.

The syntax is:

${parameter/pattern/string}

The pattern is a glob or extglob pattern. The parameter is expanded and a search carried out for the longest match with pattern, which is replaced with string.

If there is no string the matched pattern is deleted. It is acceptable to simplify the expression if that is so:

${parameter/pattern}

The first character of pattern affects the search:

  • '/' - all matches of pattern are replaced by string
  • '#' - must match at the beginning of the expanded value of parameter
  • '%' - must match at the end of the expanded value of parameter

If the pattern needs to match one of these characters then it can be escaped by preceding it with a backslash.

Examples using simple variables

Using an old phrase which was used as a typing exercise we simply replace the word 'men' by 'people':

$ phrase='Now is the time for all good men to come to the aid of the party'
$ echo "${phrase/men/people}"
Now is the time for all good people to come to the aid of the party

Using '/' as the first character of pattern we replace all occurrences of 'the' by 'THE':

$ echo "${phrase//the/THE}"
Now is THE time for all good men to come to THE aid of THE party

Using an extglob pattern 'the' and 'to' are replaced by 'X':

$ shopt -s extglob
$ echo "${phrase//@(the|to)/X}"
Now is X time for all good men X come X X aid of X party

Unfortunately it is not possible to vary the replacement string depending on the match. It would be necessary to write something more complex such as a loop to achieve this. See below for an example script which performs such a task.

Replace the 'N' followed by two letters at the start of the string by 'XXX':

$ echo "${phrase/#N??/XXX}"
XXX is the time for all good men to come to the aid of the party

Matching a pattern which starts with '/', '#', '%' requires escaping the leading character:

$ p2='#abc#'
$ echo "${p2/\#}"
abc#
$ echo "${p2/%#}"
#abc

Note that in the second case, because the '#' is not the first character it does not need to be escaped.

Examples using arrays

If the parameter is an array expression using '@' or '*' as an index then the substitution operation is applied to each member of the array in turn.

Here we declare an indexed array whose elements are the words from the example phrase above. We replace the first letter of each element by 'X':

$ declare -a words=( $phrase )
$ echo "${words[@]/?/X}"
Xow Xs Xhe Xime Xor Xll Xood Xen Xo Xome Xo Xhe Xid Xf Xhe Xarty

Here the last letter of each element is replaced by 'X':

$ echo "${words[@]/%?/X}"
NoX iX thX timX foX alX gooX meX tX comX tX thX aiX oX thX partX

If the pattern consists of '#' or '%' on its own it is possible to add text to each element at the start or the end. Here we first add '=> ' to the start of each element, then ' <=' to the end:

$ echo "${words[@]/#/=> }"
=> Now => is => the => time => for => all => good => men => to => come => to => the => aid => of => the => party
$ echo "${words[@]/%/ <=}"
Now <= is <= the <= time <= for <= all <= good <= men <= to <= come <= to <= the <= aid <= of <= the <= party <=

It is possible for the string part to be a reference to another variable (or even a command substitution):

$ echo "${words[@]/#/${words[1]} }"
is Now is is is the is time is for is all is good is men is to is come is to is the is aid is of is the is party

However the value is derived once before the multiple substitutions begin, since this statement is not a script and is executed internally by Bash:

$ echo "${words[@]/#/$RANDOM }"
9559 Now 9559 is 9559 the 9559 time 9559 for 9559 all 9559 good 9559 men 9559 to 9559 come 9559 to 9559 the 9559 aid 9559 of 9559 the 9559 party

The 'RANDOM' variable was accessed only once and its value used repeatedly.

Changing case

The final parameter expansion we’ll look at modifies the case of alphabetic characters in parameter. The syntax definition (from the GNU Bash manual) are:

${parameter^pattern}
${parameter^^pattern}

${parameter,pattern}
${parameter,,pattern}

The first pair change to upper case, and the second pair to lower case. It is important to understand the way this expansion expression is described in the manual:

Each character in the expanded value of parameter is tested against pattern, and, if it matches the pattern, its case is converted. The pattern should not attempt to match more than one character.

This means that the pattern matches only one character, not a word or similar.

Also, there’s this:

… the '^' and ',' expansions match and convert only the first character in the expanded value.

This can catch the unwary - it certainly caught me until I read the description properly!

Where the '^' or ',' is doubled the case changing operation is performed on every matching character, otherwise it operates on the first character only.

If the pattern is omitted, it is treated like a '?', which matches every character.

If the parameter is an array variable with '@' or '*' as a subscript then the case changing operations are carried out on each element.

Examples using simple variables

Using the phrase from earlier, we can alter the case of every vowel:

$ echo "${phrase^^[aeiou]}"
NOw Is thE tImE fOr All gOOd mEn tO cOmE tO thE AId Of thE pArty

Note that the following expression actually does nothing, as should be apparent from the second extract from the manual above:

$ echo "${phrase^[aeiou]}"
Now is the time for all good men to come to the aid of the party

Contrary to what you might expect, the first vowel is not converted. Instead the pattern is compared with the first letter 'N' which it doesn’t match - so nothing is done.

Examples using arrays

If we work with the array built earlier and use the vowel pattern:

$ echo "${words[@]^[aeiou]}"
Now Is the time for All good men to come to the Aid Of the party

Now, the pattern has matched any array element that starts with a vowel and has made that leading vowel upper case.

The next example operates on all vowels in each element to give the same result as earlier when working with the variable called 'phrase':

$ echo "${words[@]^^[aeiou]}"
NOw Is thE tImE fOr All gOOd mEn tO cOmE tO thE AId Of thE pArty

This one operates on all non-vowels (consonants):

$ echo "${words[@]^^[^aeiou]}"
NoW iS THe TiMe FoR aLL GooD MeN To CoMe To THe aiD oF THe PaRTY

Don’t be tempted to try something like the following:

$ echo "${words[@]^^@(good|men)}"
Now is the time for all good men to come to the aid of the party

Remember the description in the manual: The pattern should not attempt to match more than one character.

Examples

Example 1

I have included a downloadable script bash18_ex1.sh which shows a way of transforming individual words in text the "hard way":

#!/bin/bash

#-------------------------------------------------------------------------------
# Example 1 for Bash Tips show 18: transforming words in a string
#
# This is a contrived and overly complex example to show that replacing
# selected words in a phrase by different words is a non-trivial exercise!
#-------------------------------------------------------------------------------

#
# Enable extglob
#
shopt -s extglob

#
# What we'll work on and where we'll store the transformed version
#
phrase='Now is the time for all good men to come to the aid of the party'
newphrase=

#
# How to transform words in the phrase and a place to store the keys
#
declare -A transform=([good]='bad' [men]='people' [party]='Community')
declare -a keys

#
# Build an extglob pattern from the keys of the associative array
#
keys=( ${!transform[@]} )
targets="${keys[*]/%/|}"     # Each key is followed by a '|'
targets="${targets%|}"       # Strip the last '|'
targets="${targets// }"      # Some spaces got in there too
pattern="@(${targets})"      # Make the pattern at last

#
# Go word by word; if a word matches the pattern replace it by what's in the
# 'transform' array. Because the pattern has been built from the array keys
# we'll never have the case where a word doesn't have a transformation.
#
for word in $phrase; do
    if [[ $word == $pattern ]]; then
        word=${transform[$word]}
    fi
    newphrase+="$word "
done
echo "$newphrase"

exit

I have put a lot of comments into the script, but since it uses many of the actions that we have looked at in this series I hope you’ll be able to understand it with no trouble.

The principle is that just by adding the words to be matched and their replacements into the array 'transform' will cause the transformation to happen, because (in this case) the script has generated an extglob pattern from the keys.

Running the script generates the following output:

Now is the time for all bad people to come to the aid of the Community 

Example 2

There are other ways of doing this of course; for example the loop could just check if the current word is in the 'transform' array and replace the word if so or leave the word alone if not. However, there is no explicit "does key X exist in this associative array" feature in Bash so it’s not obvious.

I have included a second downloadable script bash18_ex2.sh which shows a simpler way of transforming individual words in text. This one uses the '-v varname' operator which we looked at in show 2659:

#!/bin/bash

#-------------------------------------------------------------------------------
# Example 2 for Bash Tips show 18: transforming words in a string, second
# simpler method that depends on the '-v' operator
#-------------------------------------------------------------------------------

#
# What we'll work on and where we'll store the transformed version
#
phrase='Now is the time for all good men to come to the aid of the party'
newphrase=

#
# How to transform words in the phrase
#
declare -A transform=([good]='bad' [men]='people' [aid]='assistance'
    [party]='Community')

#
# Go word by word; if an element with the word as a key exists in the
# 'transform' array replace the word by what's there.
#
for word in $phrase; do
    if [[ -v transform[$word] ]]; then
        word=${transform[$word]}
    fi
    newphrase+="$word "
done
echo "$newphrase"

exit

The '-v varname' operator returns true if the shell variable varname is set (has been assigned a value). Note that the script used '-v transform[$word]' - the name of the array with a subscript.

Running the script (in which the 'transform' array is very slightly different) generates the following output:

Now is the time for all bad people to come to the assistance of the Community