<article>
<h2 id="overview">Overview</h2>
<p>Recently Ken Fallon did a show on HPR, <a
href="https://hackerpublicradio.org/eps/hpr3962/index.html">number
3962</a>, in which he used a <a
href="https://www.gnu.org/software/bash/manual/html_node/Pipelines.html">Bash
pipeline</a> of multiple commands feeding their output into a
<code>while</code> loop. In the loop he processed the lines produced by
the pipeline and used what he found to download audio files belonging to
a series with <code>wget</code>.</p>
<p>This was a great show and contained some excellent advice, but the
use of the format:</p>
<pre><code>pipeline | while read variable; do ...</code></pre>
<p>reminded me of the "gotcha" I mentioned in my own <a
href="https://hackerpublicradio.org/eps/hpr2699/index.html">show
2699</a>.</p>
<p>I thought it might be a good time to revisit this subject.</p>
<h2 id="so-whats-the-problem">So, what's the problem?</h2>
<p>The problem can be summarised as a side effect of pipelines.</p>
<h3 id="what-are-pipelines">What are pipelines?</h3>
<p>Pipelines are an amazingly useful feature of Bash (and other shells).
The general format is:</p>
<pre><code>command1 | command2 ...</code></pre>
<p>Here <code>command1</code> runs in a subshell and produces output (on
its <em>standard output</em>) which is connected via the pipe symbol
(<code>|</code>) to <code>command2</code> where it becomes its
<em>standard input</em>. Many commands can be linked together in this
way to achieve some powerful combined effects.</p>
<p>A very simple example of a pipeline might be:</p>
<pre><code>$ printf &#39;World&bsol;nHello&bsol;n&#39; | sort
Hello
World</code></pre>
<p>The <code>printf</code> command (≡<code>'command1'</code>) writes two
lines (separated by newlines) on <em>standard output</em> and this is
passed to the <code>sort</code> command's <em>standard input</em>
(≡<code>'command2'</code>) which then sorts these lines
alphabetically.</p>
<p>Commands in the pipeline can be more complex than this, and in the
case we are discussing we can include a loop command such as
<code>while</code>.</p>
<p>For example:</p>
<pre><code>$ printf &#39;World&bsol;nHello&bsol;n&#39; | sort | while read line; do echo &quot;($line)&quot;; done
(Hello)
(World)</code></pre>
<p>Here, each line output by the <code>sort</code> command is read into
the variable <code>line</code> in the <code>while</code> loop and is
written out enclosed in parentheses.</p>
<p>Note that the loop is written on one line. The semi-colons are used
instead of the equivalent newlines.</p>
<h3 id="variables-and-subshells">Variables and subshells</h3>
<p>What if the lines output by the loop need to be numbered?</p>
<pre><code>$ i=0; printf &#39;World&bsol;nHello&bsol;n&#39; | sort | while read line; do ((i++)); echo &quot;$i) $line&quot;; done
1) Hello
2) World</code></pre>
<p>Here the variable <code>'i'</code> is set to zero before the
pipeline. It could have been done on the line before of course. In the
<code>while</code> loop the variable is incremented on each iteration
and included in the output.</p>
<p>You might expect <code>'i'</code> to be 2 once the loop exits but it
is not. It will be zero in fact.</p>
<p>The reason is that there are two <code>'i'</code> variables. One is
created when it's set to zero at the start before the pipeline. The
other one is created in the loop as a "clone". The expression:</p>
<pre><code>((i++))</code></pre>
<p>both creates the variable (where it is a copy of the one in the
parent shell) and increments it.</p>
<p>When the subshell in which the loop runs completes, it will delete
this version of <code>'i'</code> and the original one will simply
contain the zero that it was originally set to.</p>
<p>You can see what happens in this slightly different example:</p>
<pre><code>$ i=1; printf &#39;World&bsol;nHello&bsol;n&#39; | sort | while read line; do ((i++)); echo &quot;$i) $line&quot;; done
2) Hello
3) World
$ echo $i
1</code></pre>
<p>These examples are fine, assuming the contents of variable
<code>'i'</code> incremented in the loop are not needed outside it.</p>
<p>The thing to remember is that the same variable name used in a
subshell is a different variable; it is initialised with the value of
the "parent" variable but any changes are not passed back.</p>
<h2 id="how-to-avoid-the-loss-of-changes-in-the-loop">How to avoid the
loss of changes in the loop</h2>
<p>To solve this the loop needs to be run in the original shell, not a
subshell. The pipeline which is being read needs to be attached to the
loop in a different way:</p>
<pre><code>$ i=0; while read line; do ((i++)); echo &quot;$i) $line&quot;; done &lt; &lt;(printf &#39;World&bsol;nHello&bsol;n&#39; | sort)
1) Hello
2) World
$ echo $i
2</code></pre>
<p>What is being used here is <a
href="https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html"><em>process
substitution</em></a>. A list of commands or pipelines are enclosed with
parentheses and a <code>'less than'</code> sign prepended to the list
(with no intervening spaces). This is functionally equivalent to a
(temporary) file of data.</p>
<p>The <em>redirection</em> feature allows for data being read from a
file in a loop. The general format of the command is:</p>
<pre><code>while read variable
    do
       # Use the variable
    done &lt; file</code></pre>
<p>Using process substitution instead of a file will achieve what is
required if computations are being done in the loop and the results are
wanted after it has finished.</p>
<h3 id="beware-of-this-type-of-construct">Beware of this type of
construct</h3>
<p>The following one-line command sequence looks similar to the version
using process substitution, but is just another form of pipeline:</p>
<pre><code>$ i=0; while read line; do echo $line; ((i++)); done &lt; /etc/passwd | head -n 5; echo $i
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
0</code></pre>
<p>This will display the first 5 lines of the file but does it by
reading and writing the entire file and only showing the first 5 lines
of what is written by the loop.</p>
<p>What is more, because the <code>while</code> is in a subshell in a
pipeline changes to variable <code>'i'</code> will be lost.</p>
<h2 id="advice">Advice</h2>
<ul>
<li><p>Use the <em>pipe-connected-to-loop</em> layout if you're aware of
the pitfalls, but will not be affected by them.</p></li>
<li><p>Use the <em>read-from-process-substitution</em> format if you
want your loop to be complex and to read and write variables in the
script.</p></li>
<li><p>Personally, I always use the second form in scripts, but if I'm
writing a temporary one-line thing on the command line I usually use the
first form.</p></li>
</ul>
<h2 id="tracing-pipelines-advanced">Tracing pipelines (advanced)</h2>
<p>I have always wondered about processes in Unix. The process you log
in to, normally called a <em>shell</em> runs a <em>command language
interpreter</em> that executes commands read from the standard input or
from a file. There are several such interpreters available, but we're
dealing with <em>bash</em> here.</p>
<p>Processes are fairly lightweight entities in Unix/Linux. They can be
created and destroyed quickly, with minimal overhead. I used to work
with Digital Equipment Corporation's <em>OpenVMS</em> operating system
which also uses processes - but these are much more expensive to create
and destroy, and therefore slow and less readily used!</p>
<p>Bash pipelines, as discussed, use <em>subshells</em>. The description
in the Bash man page says:</p>
<blockquote>
<p>Each command in a multi-command pipeline, where pipes are created, is
executed in a subshell, which is a separate process.</p>
</blockquote>
<p>So a <em>subshell</em> in this context is basically another child
process of the main login process (or other parent process), running
Bash.</p>
<p>Processes (subshells) can be created in other ways. One is to place a
collection of commands in parentheses. These can be simple Bash
commands, separated by semi-colons, or pipelines. For example:</p>
<pre><code>$ (echo &quot;World&quot;; echo &quot;Hello&quot;) | sort
Hello
World</code></pre>
<p>Here the strings <code>"World"</code> and <code>"Hello"</code>, each
followed by a newline are created in a subshell and written to standard
output. These strings are piped to <code>sort</code> and the end result
is as shown.</p>
<p>Note that this is different from this example:</p>
<pre><code>$ echo &quot;World&quot;; echo &quot;Hello&quot; | sort
World
Hello</code></pre>
<p>In this case <code>"World"</code> is written in a separate command,
then <code>"Hello"</code> is written to a pipeline. All
<code>sort</code> sees is the output from the second <code>echo</code>,
which explains the output.</p>
<p>Each process has a unique numeric id value (the <em>process id</em>
or <em>PID</em>). These can be seen with tools like <code>ps</code> or
<code>htop</code>. Each process holds its own PID in a Bash variable
called <code>BASHPID</code>.</p>
<p>Knowing all of this I decided to modify Ken's script from <a
href="https://hackerpublicradio.org/eps/hpr3962/index.html">show
3962</a> to show the processes being created - mainly for my interest,
to get a better understanding of how Bash works. I am including it here
in case it may be of interest to others.</p>
<pre><code>#!/bin/bash

series_url=&quot;https://hackerpublicradio.org/hpr_mp3_rss.php?series=42&amp;full=1&amp;gomax=1&quot;
download_dir=&quot;./&quot;

pidfile=&quot;/tmp/hpr3962.sh.out&quot;
count=0

echo &quot;Starting PID is $BASHPID&quot; &gt; $pidfile

(echo &quot;[1] $BASHPID&quot; &gt;&gt; &quot;$pidfile&quot;; wget -q &quot;${series_url}&quot; -O -) |&bsol;
    (echo &quot;[2] $BASHPID&quot; &gt;&gt; &quot;$pidfile&quot;; xmlstarlet sel -T -t -m &#39;rss/channel/item&#39; -v &#39;concat(enclosure/@url, &quot;→&quot;, title)&#39; -n -) |&bsol;
    (echo &quot;[3] $BASHPID&quot; &gt;&gt; &quot;$pidfile&quot;; sort) |&bsol;
    while read -r episode; do

        [ $count -le 1 ] &amp;&amp; echo &quot;[4] $BASHPID&quot; &gt;&gt; &quot;$pidfile&quot;
        ((count++))

        url=&quot;$( echo &quot;${episode}&quot; | awk -F &#39;→&#39; &#39;{print $1}&#39; )&quot;
        ext=&quot;$( basename &quot;${url}&quot; )&quot;
        title=&quot;$( echo &quot;${episode}&quot; | awk -F &#39;→&#39; &#39;{print $2}&#39; | sed -e &#39;s/[^A-Za-z0-9]/_/g&#39; )&quot;
        #wget &quot;${url}&quot; -O &quot;${download_dir}/${title}.${ext}&quot;
    done

echo &quot;Final value of &bsol;$count = $count&quot;
echo &quot;Run &#39;cat $pidfile&#39; to see the PID numbers&quot;</code></pre>
<p>The point of doing this is to get information about the pipeline
which feeds data into the <code>while loop</code>. I kept the rest
intact but commented out the <code>wget</code> command.</p>
<p>For each component of the pipeline I added an <code>echo</code>
command and enclosed it and the original command in parentheses, thus
making a multi-command process. The <code>echo</code> commands write a
fixed number so you can tell which one is being executed, and it also
writes the contents of <code>BASHPID</code>.</p>
<p>The whole thing writes to a temporary file
<code>/tmp/hpr3962.sh.out</code> which can be examined once the script
has finished.</p>
<p>When the script is run it writes the following:</p>
<pre><code>$ ./hpr3962.sh
Final value of $count = 0
Run &#39;cat /tmp/hpr3962.sh.out&#39; to see the PID numbers</code></pre>
<p>The file mentioned contains:</p>
<pre><code>Starting PID is 80255
[1] 80256
[2] 80257
[3] 80258
[4] 80259
[4] 80259</code></pre>
<p>Note that the PID values are incremental. There is no guarantee that
this will be so. It will depend on whatever else the machine is
doing.</p>
<p>Message number 4 is the same for every loop iteration, so I stopped
it being written after two instances.</p>
<p>The initial PID is the process running the script, not the login
(parent) PID. You can see that each command in the pipeline runs in a
separate process (<em>subshell</em>), including the loop.</p>
<p>Given that a standard pipeline generates a process per command, I was
slightly surprised that the PID numbers were consecutive. It seems that
Bash optimises things so that only one process is run for each element
of the pipe. I expect that it would be possible for more processes to be
created by having pipelines within these parenthesised lists, but I
haven't tried it!</p>
<p>I found this test script quite revealing. I hope you find it useful
too.</p>
<h2 id="links">Links</h2>
<ul>
<li>Bash pipelines:
<ul>
<li><a
href="https://www.gnu.org/software/bash/manual/html_node/Pipelines.html">GNU
Bash manual: 3.2.3 Pipelines</a></li>
</ul></li>
</ul>
<ul>
<li>Bash loops:
<ul>
<li><a
href="https://www.gnu.org/software/bash/manual/html_node/Looping-Constructs.html">GNU
Bash manual: 3.2.5.1 Looping Constructs</a></li>
</ul></li>
</ul>
<ul>
<li>Bash process substitution:
<ul>
<li><a
href="https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html">GNU
Bash manual: 3.5.6 Process Substitution</a></li>
</ul></li>
</ul>
<ul>
<li>HPR shows referenced:
<ul>
<li><a
href="https://hackerpublicradio.org/eps/hpr2045/index.html">hpr2045 ::
Some other Bash tips</a></li>
<li><a
href="https://hackerpublicradio.org/eps/hpr2699/index.html">hpr2699 ::
Bash Tips - 15</a></li>
<li><a
href="https://hackerpublicradio.org/eps/hpr3962/index.html">hpr3962 ::
It's your data</a></li>
</ul></li>
</ul>
</article>
