Site Map - skip to main content - dyslexic font - mobile - text - print

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.


In-Depth Series

Bash Scripting

This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.

Some supplementary Bash tips - Dave Morriss | 2017-04-26

Some supplementary Bash tips

Pathname expansion; part 1 of 2

Expansion

As we saw in the last episode 2045 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion (seen in episode 1951)
  • Process substitution (seen in episode 2045)
  • Word splitting (seen in episode 2045)
  • Pathname expansion (this episode and the next)

This is the last topic in the (sub-) series about expansion in Bash. However, when writing the notes for this episode it became apparent that there was too much to fit into a single HPR episode. Consequently I have made it into two.

In this episode we will look at simple pathname expansion and some of the ways in which its behaviour can be controlled. In the next episode we’ll finish by looking at extended pattern matching. Both are included in the “Manual Page Extracts” section at the end of the long notes.

Long Show Notes

I have written out a moderately long set of notes about this subject and these are available here.


Audio speedup script - Dave Morriss | 2016-10-07

Audio speedup script

Back in 2015 Ken Fallon did a show (episode 1766) on how to use sox to truncate silence and speed up audio.

Inspired by this I wrote a Bash script to aid my use of the technique, which I thought I’d share with you.

I have written out detailed notes for this episode describing the script and examining how it works and these are available here http://hackerpublicradio.org/eps/hpr2135/full_shownotes.html.


Useful Bash functions - part 2 - Dave Morriss | 2016-08-15

Useful Bash functions - part 2

Overview

This is the second show about Bash functions. In this one I revisit the yes_no function from the last episode and deal with some of the deficiencies of that version.

As before it would be interesting to receive feedback on these versions of the function and would be great if other Bash users contributed ideas of their own.

Full Notes

Since the notes explaining this subject are long, they have been placed here.


Some other Bash tips - Dave Morriss | 2016-06-03

Some other Bash tips

Expansion

As we saw in the last episode 1951 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion (seen in episode 1951)
  • Process substitution
  • Word splitting
  • Pathname expansion

We will look at process substitution and word splitting in this episode but since there is a lot to cover in these subjects, we'll save pathname expansion for the next episode.

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr2045/full_shownotes.html.


Using the Incron file watching daemon - b-yeezi | 2016-04-06

Using the Incron file watching daemon

Check out the man page for incron and also this write-up by Nixcraft.

basic usage:

incrontab -e

In your editor of choice, follow this syntax:

<path-to-watch> <event mask> command

Links


Pomodoro Timer - The Evolution of a Script part deux - Nacho Jordi | 2016-03-18

The Script: http://hackerpublicradio.org/eps/hpr1990.bash


Pomodoro Timer - The Evolution of a Script (pt 1) - Nacho Jordi | 2016-03-15


Time now Ladies and Gents - Ken Fallon | 2016-01-26

In the show "hpr1943 :: HPR AudioBook Club 11.5 - Interview with David Collins-Rivera" pokey asked if there was a way to get the duration for media. The following three options springs to mind immediately.

The first option is fix_tags and was written by our own Dave Morriss.

$ date --utc --date="@$(echo $(fix_tags *mp3 *ogg 2>/dev/null | \
awk -F '\\(|\\)' '/length/ {print $2}' | \
sed 's/ sec//g' ) | \
sed 's/ /+/g' | bc )"  +"%T"
03:09:49

Next up is mediainfo which provides a lot of information on media files.

$ date -ud @$(echo $(mediainfo --full --Output=XML *mp3 *ogg | \
xmlstarlet sel -T -t -m "Mediainfo/File/track[@type='Audio']/Duration[1]" -v "." -n - | \
sed 's/.\{3\}$//') | \
sed 's/ /+/g' | bc)  +"%T"
03:09:49

The last option is to use ffprobe from the ffmpeg team.

$ date -ud @$(echo $(for i in *mp3 *ogg;\
do  \date -ud 1970-01-01T$(ffprobe -i $i 2>&1 | \
grep Duration | awk '{print $2}'| \
sed 's/,//g' ) +%s;done) | \
sed 's/ /+/g' | bc)  +"%T"
03:09:49

For complete shownote please visit http://hackerpublicradio.org/eps/hpr1952.html


Some additional Bash tips - Dave Morriss | 2016-01-25

Some additional Bash tips

Expansion

As we saw in the last episode 1903 there are seven types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion
  • Word splitting
  • Pathname expansion

There is also another, process substitution, which occurs after arithmetic expansion on systems that can implement it.

We will look at one more of these expansion types in this episode but since there is a lot to cover, we'll continue in a later episode.

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr1951_full_shownotes.html.

Audio Note

This time, in the spirit of experimentation and as a way of learning Audacity I processed my audio thus:

  • Turned the stereo tracks to mono

  • Used a Noise Gate plug-in to reduce background noise (after "training" it on some silence)

  • Performed a Truncate Silence pass to reduce the length of pauses

  • Applied a small amount of amplification

Let me know if this had any positive or negative effects on the end product.


Some further Bash tips - Dave Morriss | 2015-11-18

Some further Bash tips

Expansion

There are seven types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in the last episode 1884)
  • Tilde expansion
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution
  • Arithmetic expansion
  • Word splitting
  • Pathname expansion

We will look at some more of these in this episode but since there is a lot to cover, we'll continue in a later episode.

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr1903_full_shownotes.html.


Some more Bash tips - Dave Morriss | 2015-10-22

Some more Bash tips

We looked at Parameter Expansion back in HPR episode 1648 where we saw how Bash variables could be used, checked and edited. There are other sorts of expansions within Bash, and we'll look at one called "Brace Expansion" in this episode, which follows on from episode 1843 "Some Bash tips".

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr1884_full_shownotes.html.


Some Bash tips - Dave Morriss | 2015-08-26

Today I want to talk about three Bash commands:

  • pushd
  • popd
  • dirs

These let you change directory on a Linux system (and others which support Bash) but keep a record of where you have been in a stack structure. The stack can be viewed and manipulated with these commands as well.

I have written out a moderately long set of notes about these commands and these are available here http://hackerpublicradio.org/eps/hpr1843_full_shownotes.html.


Waking up with Windigo - Windigo | 2015-08-19

This is a quick summary of my alarm clock system, written in bash and highly unreliable.

Hardware

My preferred hardware platform is a Dell Mini 9.

Software

My alarm clock is an embarrassing combination of bash scripts and Audacious, my favorite media player. Any media player will do, as long as it's scriptable.

How It Works

There are currently two bash scripts in my crappy alarm setup. One script is called "wakeup" and the other is called "wakeup-at".

wakeup is simply a wrapper that adds some error handling around audacious. It launches audacious if it can't find an instance running already, waits five seconds for it to get itself together, and then causes it to play. It is also currently broken, so the 'launching audacious' part doesn't work. I have to manually start audacious myself. FAILURE.

wakeup script:

#!/bin/bash
audacious &

sleep 5s

audacious -p &

You've noticed that the "wakeup" script doesn't actually have any timing involved; If you want to use it as an alarm, you get to combine it with the bash "sleep" command. This is not a failure, this is by design! An example alarm:

sleep 8h; wakeup

One problem with this methodology is that it requires math, and is prone to errors. If I'm going to sleep at 10:46:33 PM and need to wake up at 7:00 AM, I need to chain sleep commands together for each unit of time:

sleep 7h; sleep 14m; sleep 27s; wakeup

Get some of that math wrong, and you wake up at the wrong time. FAILURE.

"wakeup-at" is a wrapper around "wakeup" that uses the "at" utility to schedule the wakeup script. So, instead of using multiple sleep commands, it accepts any of the time formats that at accepts:

wakeup-at 7:00 AM
wakeup-at 6:00AM 2018-02-02
wakeup-at teatime

Here is the wakeup-at script:

#!/bin/bash

## Make sure we have enough arguments
if [ $# -lt 1 ]
then
  echo "Usage: `basename $0` <time>"
  exit 1
fi

echo "$@"

## Add custom time keywords
case "$1" in
"eternaldarkness")
	echo wakeup | at 3:33 AM
	;;

## Catch-all; send all arguments to at
*)
	echo wakeup | at $@
	;;
esac

If you make a syntax error, "at" tells you about it immediately. Its only failings are what it inherits from the original "wakeup" script.


Organizing Photos with Bash - Tony Pelaez | 2015-06-15

Summary

In this episode I provide an overview of how I use bash to automate my process for orgainizing photographs on my computer.

There are two main objectives of this script:

  1. Organize photographs in a folder structure that makes sense to me, e.g. 2015/2015-05-22
  2. Allow me to back up my photographs using a variety of methods.

Download the Script

This script is hosted on Github and you can download the latest version using following command:

git clone https://gist.github.com/81e489b2a7397bb17305.git

Script


#!/bin/bash

shopt -s -o nounset

# Create variables and configure script.
declare -rx SCRIPT=${0##*/}
declare TMPDIR=/tmp/photos
declare -r CURRENTDIR=`pwd`
declare FILES=$TMPDIR/*
declare DESTINATION=/media/Tyr/Pictures/Photos
declare -r GOOGLEUSER="tnyplz@gmail.com"
declare -r  OPTSTRING="-h, -d:"
declare -r  LONGOPTSTRING="help, destination-directory, no-google-backup, sd-card, tmp-dir, no-delete, backup"
declare RESULT
declare GOOGLE_BACKUP=true
declare SD=false
declare SDDIR
declare NODELETE=false
declare S3=false

# Executable dependencies
declare -rx find="/usr/bin/find"
declare -rx gphoto2="/usr/bin/gphoto2"
declare -rx google="/usr/bin/google"
declare -rx dcraw="/usr/bin/dcraw"
declare -rx rsync="/usr/bin/rsync"
declare -rx rename="/usr/bin/rename"
declare -rx tar="/usr/bin/tar"
declare -rx s3cmd="/usr/bin/s3cmd"

# Sanity Checks
if test -z $BASH; then
    printf "$SCRIPT:$LINENO: please run this script with the BASH shell\n" >&2
    exit 192
fi
# check for find
if test ! -x $find; then
    printf "$SCRIPT:$LINENO: the $find command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for gphoto2
if test ! -x $gphoto2; then
    printf "$SCRIPT:$LINENO: the $gphoto2 command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for google
if test ! -x $google; then
    printf "$SCRIPT:$LINENO: the $google command is not available -- \
aborting\n" >&2
fi
# check for dcraw
if test ! -x $dcraw; then
    printf "$SCRIPT:$LINENO: the $dcraw command is not available -- \
aborting\n" >&2
fi
# check for rename
if test ! -x $rename; then
    printf "$SCRIPT:$LINENO: the $rename command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for rsync
if test ! -x $rsync; then
    printf "$SCRIPT:$LINENO: the $rsync command is not available -- \
aborting\n" >&2
fi
# check for tar
if test ! -x $tar; then
    printf "$SCRIPT:$LINENO: the $tar command is not available -- \
aborting\n" >&2
fi
# check for glacier-cmd
if test ! -x $s3cmd; then
    printf "$SCRIPT:$LINENO: the $s3cmd command is not available -- \
aborting\n" >&2
fi


# Check for Options
# =================

getopt -T
if [ $? -ne 4 ]; then
    printf "$SCRIPT:$LINENO: %s\n" "getopt is in compatibility mode" >&2
    exit 192
fi

RESULT=$(getopt --name "$SCRIPT" --options "$OPTSTRING" --longoptions "$LONGOPTSTRING" -- "$@")
if [ $? -gt 0 ]; then
    exit 192
fi

eval set -- "$RESULT"

while [ $# -gt 0 ]; do
    case "$1" in
    -h | --help) # show help
        printf "%s\n" "
This script helps you automate the process of downloading photos from
your camera, uploading backups to Google Picasa, and syncing the files
with a specified directory.

Dependendies:
  gphoto2
  dcraw
  googlecl
  rsync
  s3cmd

usage: $SCRIPT [options]

Options:
  -h | --help                        Show help for $SCRIPT
  --destination-directory {LOCATION} Set the location where the photos will be
                                     copied to.
  --tmp-dir {LOCATION}               Set the temporary directory where images
                                     will be downloaded to initially. The
                                     default is /tmp/photos.
  --no-google-backup                 Disable uploading low rez copies to Google
                                     Plus.
  --sd-card {LOCATION}               Set the location of the sd card.
  --no-delete                        Do not delete from temp file.
  --backup {FOLDER} {S3 BUCKET}      Create archive from folder and upload to S3.
  "
        exit 0
        ;;
    --destination-directory ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for destination. No destination given." >&2
            exit 192
        fi
        DESTINATION="$1"
        ;;
    --tmp-dir ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for tmp-dir.  No temporary directory given." >&2
            exit 192
        fi
        TMPDIR="$1"
        FILES=$TMPDIR/*
        ;;
    --no-google-backup ) shift
        GOOGLE_BACKUP=false
        ;;
    --sd-card ) shift
        SD=true
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for sd directory. No sd card directory given." >&2
            exit 192
        fi
        SDDIR="$1"
        ;;
    --no-delete ) shift
	NODELETE=true
	;;
    --backup ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for AWS Glacier Backup. Backup folder and vault must be specified."
        fi
        S3=true
        BACKUP_FOLDER="$1"
        BUCKET="$2"
        ;;
    esac
    shift
done

# Functions
# =========

# function to convert a raw image to jpg.
# input: requires the user to specify the file extention ($1).
function convert_to_jpg () {
  FILES2CONVERT=$TMPDIR/*"$1"
  for FILE in $FILES2CONVERT
  do
    FILE2BACKUP=$TMPDIR/Backup/`basename "$FILE" "$1"`'.jpg'
    if [ -e $FILE2BACKUP ]; then
        printf "$SCRIPT:$LINENO: Skipping $FILE, jpg file already exists\n"
    elif [ -e $FILE ]; then
        printf "$SCRIPT:$LINENO: Converting $FILE to $FILE2BACKUP\n"
        $dcraw -cvz -w -o 1 -q 3 "$FILE" | cjpeg -quality 80 -optimize > "$FILE2BACKUP"
    else
        printf "Did not convert $FILE\n"
    fi
  done
}

# function to resize jpeg to upload to picasa
function resize_to_thumb () {
    FILES2RESIZE=$TMPDIR/Backup/* # TODO pass this in as argument along with destination directory
    for FILE in $FILES2RESIZE
    do
        printf "$SCRIPT:$LINENO: Creating thumbnail for $FILE..."
        convert $FILE -resize 2048x2048 $TMPDIR/Backup/Upload/`basename "$FILE" ".jpg"`'_thumb.jpg'
        printf "done\n"
    done
}

# function to import photos
function import_photos () {
    printf "$SCRIPT:$LINENO: Importing Photos\n"
    if $SD; then
        cp -p "$SDDIR"/* .
    else
        $gphoto2 --quiet --get-all-files
    fi
}

# function to remove spaces in file names
function remove_spaces () {
    $find $1 -depth -name "* *" -execdir $rename 's/ /_/g' "{}" \;
}

# function to sort images into direcotries based on date.
# input: directory to sort ($1)
#        directory to sort into ($2)
function sort_images () {
    SORTDIR=$2'/Sorted/'
    for FILE in $1
    do
        printf "$SCRIPT:$LINENO: Sorting $FILE\n"
        DATEDIR=$SORTDIR`date -r "$FILE" +%Y`'/'`date -r "$FILE" +%Y-%m-%d`
        mkdir -p $DATEDIR
        cp "$FILE" $DATEDIR/
    done
}

# function create archive and upload to AWS S3
# input: directory to create an archive for ($1)
#        s3 bucket name ($2)
function archive_folder () {
    ARCHIVE=$TMPDIR/$(basename $1).tar.bz2
    printf "$SCRIPT:$LINENO: archiving $ARCHIVE\n"
    $tar -cvjf $ARCHIVE $1
    $s3cmd put $ARCHIVE $2
}

# Create temporary directory
mkdir -p $TMPDIR
cd $TMPDIR

# Create AWS Glacier archive
if $S3; then
    archive_folder $BACKUP_FOLDER $BUCKET
    cd $CURRENTDIR
    if [ $NODELETE = false ]; then
        rm -rf $TMPDIR
    fi
    exit 0
fi

# Import files from camera
import_photos
printf "$SCRIPT:$LINENO: Importing Photos Done!\n"

# Remove Spaces in Filenames
remove_spaces $TMPDIR

#Convert all files to lower case
printf "$SCRIPT:$LINENO: Converting Photos to Lower Case.\n"
for FILE in *
do
    f=`echo $FILE | tr '[:upper:]' '[:lower:]'`
    mv "$FILE" "$f"
done
printf "$SCRIPT:$LINENO: Converting Photos to Lower Case Done!\n"

# Sort files
sort_images "$FILES" "$TMPDIR"
printf "$SCRIPT:$LINENO: Sorting Images Done!\n"

# Create backup jpgs and upload them to Picassa
mkdir -p $TMPDIR/Backup
cp $TMPDIR/*.jpg $TMPDIR/Backup/

convert_to_jpg ".nef"
convert_to_jpg ".nrw"
mkdir -p $TMPDIR/Backup/Upload
resize_to_thumb

if $GOOGLE_BACKUP; then
    # Upload jpgs to Picassa
    # Requires that you authorize googlecl through the web browser.
    $google picasa create --user $GOOGLEUSER --title "Backup "`date +%Y-%m` $TMPDIR/Backup/Upload/*
fi

# Copy files to final locations
$rsync -ravv $TMPDIR/Sorted/ $DESTINATION # TODO test to make sure destination works

cd $CURRENTDIR

# Remove temp folder
if [ $NODELETE = false ]; then
    rm -rf $TMPDIR
fi

printf "$SCRIPT:$LINENO: Processing Complete!\n"
exit 0


Useful Bash functions - Dave Morriss | 2015-04-28

Overview

I enjoy writing Bash scripts to solve various problems. In particular I have a number of scripts I use to manage the process of preparing a show for HPR, which I am developing at the moment.

My more complex Bash scripts use a lot of functions to perform the various tasks, and, in the nature of things, some of these functions can be of use in other scripts and are shared between them.

I thought I would share some of these functions with HPR listeners in the hopes that they might be useful. It would also be interesting to receive feedback on these functions and would be great if other Bash users contributed ideas of their own.

Full Notes

Since the notes explaining this subject are long, they have been placed here: http://hackerpublicradio.org/eps/hpr1757_full_shownotes.html, and an experimental ePub version is available here: http://hackerpublicradio.org/eps/hpr1757_full_shownotes.epub.

  1. Bash Support Vim plugin: http://www.vim.org/scripts/script.php?script_id=365
  2. HPR episode Bash parameter manipulation: http://hackerpublicradio.org/eps.php?id=1648
  3. How to write functions (from The Linux Documentation Project):
  4. Download the pad and yes_no functions: http://hackerpublicradio.org/eps/hpr1757_functions.sh

Windows Remote Desktop on GNU/Linux - Ken Fallon | 2015-02-20

Recorded using Easy Voice Recorder Pro

I wrote a bash script to connect to various different windows servers from my GNU/Linux desktops. I had a few different requirements:

  • I should be able to call it based on hostname.
  • All windows should be 90% smaller than my screen.
  • It should map my keyboard.
  • It should map my local disk.
  • It should quickly timeout if the port is not available.

You can get the full script here, but let’s walk through it:

The first line calls bash and then gets the server name from the symlink that is calling the script. The port is set as “3389”, but you can change that if you like.

#!/bin/bash
SERVER=`basename $0`
PORT="3389"

The next few lines finds the smallest vertical and horizontal sizes, even if you are running multiple screens. Then it calculates 90% of that to use as the size.

h=$(echo "scale=0;(($(xrandr | grep '*+' | sed 's/x/ /g' | awk '{print $1}' | sort -n | head -1 )/100)*90)" | bc)
v=$(echo "scale=0;(($(xrandr | grep '*+' | sed 's/x/ /g' | awk '{print $2}' | sort -n | head -1 )/100)*90)" | bc)
SIZE=${h}x${v}

Next we set the default username and password. I have it ask me for my password but I put it in here as an example.

PASSWORD='defaultpassword'
USERNAME='administrator'
WORKGROUP='workgroup'

In some cases the credentials may be different, so I have a case statement that will cycle through the servers and apply the differences. Depending on your naming schemes you may be able to use regular expressions here to filter out groups of servers.

case "${SERVER}" in
  *server*) echo "Server ${SERVER}"
    PASSWORD='work_password'
    USERNAME='administrator'
    WORKGROUP='WORKGROUP'
    ;;
 
  *colo*) echo "Server ${SERVER}"
    PASSWORD='colo_server_password'
    USERNAME='administrator'
    WORKGROUP='COLODOMAIN'
    ;;
     
  some_server ) echo "Server ${SERVER}"
    PASSWORD='some_server_password'
    USERNAME='some_server_password'
    ;;
  *) echo "No match for ${SERVER}, using defaults"
    ;;
esac

Next we use an inbuilt bash command to see if a remote port is open and timeout after one second.

timeout 1 bash -c "echo >/dev/tcp/${SERVER}/${PORT}"

I used to connect to rdp using the program rdesktop, but it is now of limited value due to the fact that there are many open bugs that are not getting fixed. Bugs such as Bug 1075697 - rdesktop cannot connect to systems using RDP version 6 or newer and Bug 1002978 - Failed to negotiate protocol, retrying with plain RDP . I then switch to using xfreerdp. This is the client that is behind remmina.

You can use xfreerdp /kbd-list to get a list of the available keyboard layouts.

if [ $? -eq 0 ]; then
  echo "${SERVER}:${PORT} is open"
  xfreerdp /v:${SERVER} /size:${SIZE} /kbd-type:0x00000409 /t:${SERVER} /d:${WORKGROUP} /u:${USERNAME} /p:${PASSWORD} /a:drive,pc,/ /cert-ignore &
else
  echo "${SERVER}:${PORT} is closed"
fi

Next you will need to be sure that your host names are available, either in dns or in your /etc/hosts/ file. For example:

10.1.0.1 server1
10.1.0.2 server2
10.1.0.3 server3
10.2.0.1 coloserver1
10.2.0.2 coloserver2
10.2.0.3 coloserver3
192.168.1.1 some_server

Edit the script to your liking and then put it into your a directory in your path, possibly /usr/local/bash or ~/bin/. You can then make symbolic links to the servers to the bash script, also in a directory in your path, using the command:

ln -s /usr/local/bash/rdp.bash ~/bin/some_server
chmod +x ~/bin/some_server

Which links the global rdp.bash script to your personal symlink, and makes it executable.

All that you need to do then is type the name of the server and a rdp screen should pop up.

In our example:

$ some_server

From there your Windows Server session should pop up.


Bash parameter manipulation - Dave Morriss | 2014-11-26

Bash parameter manipulation

I'm a great fan of using the Linux command line and enjoy writing shell scripts using the Bash shell.

  • BASH (or more usually Bash or bash) is the name of a Unix shell. The name stands for Bourne Again SHell, which is a play on words. Bash is an extension of the shell originally written by Stephen Bourne in 1978, usually known as SH.

  • Bash was written as part of the GNU Project which forms part of the Linux Operating System.

  • A shell is the part of the operating system that interprets commands, more commonly known as the command line.

  • A knowledge of Bash is very helpful if you would like to be able to use the power of the command line. It is also the way to learn how to build Bash scripts for automating the tasks you need to perform.

In this episode we look at what parameters are in Bash, and how they can be created and manipulated. There are many features in Bash that you can use to do this, but they are not easy to find.

As I was learning my way around Bash it took me a while to find these. Once I had found them I wanted to make a "cheat sheet" I could stick on the wall to remind me how to do things. I am sharing the result of this process with you.

The version of Bash which I used for this episode is 4.3.30(1)-release

The full notes for this episode are to be found here: http://hackerpublicradio.org/eps/hpr1648_full_notes.html


thebestofyoutube.com download script - Ken Fallon | 2014-01-24

In episode "Thu 2013-12-19: hpr1404 Editing pre-recorded audio in Audacity" I walked you through editing a podcast, by the magic of editing this is been posted after the other show has aired. The plan here is to get people to share their useful hacks to show how elegant, or in my case ugly, code can be. As Knightwise says "Getting technology to work for you."™
Feel free to share your own hacks with us.

http://hackerpublicradio.org/eps.php?id=1404
http://hackerpublicradio.org/eps/hpr1430-downloader.bash.txt


#!/bin/bash
# Downloads videos from youtube based on selection from http://thebestofyoutube.com
# (c) Ken Fallon http://kenfallon.com
# Released under the CC-0

maxtodownload=10
savepath="/mnt/media/Videos/tv/youtube/bestofyoutube"
savedir="${savepath}/$(\date -u +%Y-%m-%d_%H-%M-%SZ_%A)"
mkdir -p ${savedir}
logfile="${savepath}/downloaded.log"

# Gather the list
seq 1 ${maxtodownload} | while read videopage;
do 
  thisvideolist=$(wget --quiet "http://bestofyoutube.com/index.php?page=${videopage}" -O - | 
  grep 'www.youtube.com/embed/' | 
  sed 's#^.*www.youtube.com/embed/##' | 
  awk -F '"|?' '{print "http://www.youtube.com/watch?v="$1}')
  for thisvideo in $(echo $thisvideolist);
  do 
    if [ "$( grep "${thisvideo}" "${logfile}" | wc -l )" -eq 0 ];
    then
      echo "Found the new video ${thisvideo}"
      echo ${thisvideo} >> ${logfile}_todo
    else
      echo "Already downloaded ${thisvideo}"
    fi
  done
done

# Download the list
if [ -e ${logfile}_todo ];
then
  tac ${logfile}_todo | youtube-dl --batch-file - --ignore-errors --no-mtime --restrict-filenames \
    --max-quality --format mp4 --write-auto-sub -o ${savedir}'/%(autonumber)s-%(title)s-%(id)s.%(ext)s'
  cat ${logfile}_todo >> ${logfile}
  rm ${logfile}_todo
fi


Recording Terrestrial Radio with bash scripts and cron jobs - Jon Kulp | 2013-05-14

In this episode I talk about my solution for capturing terrestrial radio so that I can listen to it at my own convenience. I use a bash script, cron jobs, and the streamripper package. here are some links to things I mentioned in the podcast.

Jezra's command-line audio player sap (simple audio player): http://www.jezra.net/projects/sap

Streamripper: http://streamripper.sourceforge.net/

Radio station KRVS 88.7 FM, Lafayette, Louisiana, USA http://krvs.org/

And you can see the whole radio-recording script here: http://jonathankulp.org/archives/647


My Magnatune Downloader - Dave Morriss | 2013-03-14

The Problem

I'm a fan of Magnatune (http://magnatune.com/) and have been buying music from them for 7 or 8 years. The Magnatune website itself is good for exploring and downloading, and interfaces for browsing and purchasing are available in a number of players on Linux. I have direct experience of:

  • Amarok: allows you to browse, purchase, examine artist information and album details.
  • Rhythmbox: the plugin, which used to allow browsing and purchasing, is currently unavailable, but is apparently due to return soon.
  • Gnome Music Player Client: (a front-end to the Music Player Daemon, mpd) offers a Magnatune browser plugin
  • Magnatune Web 2.0 player: a web-based tool which will browse, play and download Magnatune music.
  • Magnatune Android player: a fairly basic browser and player for Android 2.0 and up.

The Magnatune Web 2.0 player is the best of the bunch as far as I am concerned, particularly since it allows me to explore the music collection whilst listening to streamed music at the same time. However, none of these interfaces provide me with exactly what I want in terms of the download process, so I decided to write my own.

The Plan

I currently host my music on my HP Proliant microserver, share it across the home network, and play it with the Music Player Daemon (http://sourceforge.net/projects/musicpd/) on my desktop system. I normally keep the album cover image, artwork and related material in the same directory as the album itself, and I want to be able to save all files in their appropriate places automatically.

Magnatune provides an API which is documented at http://download.magnatune.com/info/api, though this information is only available to members. Data is available in several formats: XML, SQlite and MySQL.

Design

I didn't want to launch into building a full-blown application, especially since I only needed a downloader, so I decided to create a collection of Bash and Perl scripts.

I decided to use the XML data organised by album. This is updated on about a weekly or two weekly basis, and there is a signalling mechanism through a downloadable file containing a checksum. When this changes the large data file has changed and can be downloaded. At the time of writing I simply run this by hand when I receive an email alert from Magnatune.

Magnatune uses an unique key made from the artist and album names which it refers to as the SKU (Stock Keeping Unit) or albumsku. They use this as an URL component and in XML tags. I use it to identify the stuff I download and to keep a simple inventory.

I decided to write some basic scripts:

  • To download the catalogue
  • To extract information from the catalogue
  • To download an album
  • To unpack the downloaded items into the target directory

I wanted to learn more about manipulating XML data, so I decided to use XSL, the Extensible Stylesheet Language. This lets you define stylsheets for XML data, including ways of identifying XML components with XPath and of transforming XML with XSLT.

I have included a number of links to the resources I used in the shownotes.

Repository

I have placed all of the scripts, their associated files, and HTML and PDF README files (extended shownotes) in a GitHub repository. This can be browsed at https://github.com/davmo/magnatune-downloader or, if a copy is required it can be obtained with the command:

  git clone https://github.com/davmo/magnatune-downloader.git

This makes a local git repository containing a copy of all of the files in the current directory.

Note: The code was originally hosted on Gitorious (https://gitorious.org/magnatune-downloader), but with the demise of this service it has been moved to GitHub and the details above have been updated.

Scripts

  • update_albums: a Bash script to download a new version of the album catalogue, as a bzipped XML file, if it is different from the current version. It generates a summary of the catalogue for simple searching using XSLT.
  • report_albumsku: a Bash script to take a SKU code and look up the album details in the XML file.
  • get_album: a Bash script to download an album, cover images and artwork. It takes the SKU as an argument and uses it to make an URL for an XML file which points at all of the components, and this is downloaded (with authentication). The script then parses this file to get the necessary URLs for downloading. I only use the OGG format but it could easily collect any or all formats available from Magnatune. The script records the fact that this particular SKU code has been downloaded so that it isn't collected again in error. All downloaded files are given names beginning with the SKU code and are stored for the installation phase.
  • install_download: a Perl script which unpacks the downloaded zip file to its final destination then adds the cover images and artwork to the same place. I used Perl because it allowed me to query the zip file to determine the name of the directory that was going to be created.

Further Developments

I have added further scripts to this system since I created it. I have one that synchronises the music files from my workstation to the server, and two that give me a simple wish-list or queue functionality.

Since I have a 200GB download limit per month on my broadband contract I try not to download music too often and avoid contention with the rest of the family. My queueing system is used to keep a list of stuff I'd like to buy from Magnatune, and I simply feed the top element from the queue into my download script every week or so.

In the future I expect to be refining all of these scripts and making them less vulnerable to errors. For example, I have found a few cases where Magnatune's XML is not valid and this causes the xsltproc tool to fail. I'd like to be able to recover from such errors more elegantly than I'm doing now.

At some point I may well be tempted to consolidate all of the current functions into a single Perl script.

Disclaimer

I have no connections to Magnatune other than being a contented customer.

Links


What I do with bash scripts - Jon Kulp | 2013-03-05

In this episode I talk about the way I use shell scripting on a day-to-day basis. I am not employed in a technical field, so the fact that I use shell scripts at all surprises most people. I am just a music history professor with an enthusiasm for Linux and free software. Although I have dabbled a bit with Python, I don't feel nearly as comfortable with Python as I do with bash, so all of the scripts I mention in this episode are written for bash.

Here are links to blog posts about some of the scripts mentioned in the show.

markdown2latex: http://jonathankulp.org/archives/570

Cowsay stuff: http://jonathankulp.org/archives/346

"stick" scp script: http://jonathankulp.org/archives/441

MyIP: http://jonathankulp.org/archives/620


Bash Scripting: Episode 2 Command Line Basics - Ken Fallon | 2010-11-17

In the second installment Ken resolves to not do any work and so get's permission from Chess Griffin to reuse extracts from Linux Reality Episode 14 - Command Line Basics May 17, 2006 http://www.linuxreality.com/archives.php#14 Shownotes can be found at http://hackerpublicradio.org/shownotes/hpr0598.html

Introduction to bash scripting - Ken Fallon | 2010-08-11

http://en.wikipedia.org/wiki/Bourne_shell
http://en.wikipedia.org/wiki/Command-line_interface

A list of "Hello World" programs in many different computer languages: 
http://www.roesler-ac.de/wolfram/hello.htm 

For Windows:
Editor: http://notepad-plus-plus.org/
Bash (and more): http://x.cygwin.com/ 
(run setup, and selecting the 'xinit' package from the 'X11' category.)

$ echo '#!/bin/bash' > hello.bash
$ echo "echo hello world" >> hello.bash

$ cat hello.bash 
#!/bin/bash
echo hello world

$ chmod +x hello.bash

$ ./hello.bash
hello world

feedback-(a)-kenfallon.com
More information http://www.kenfallon.com

Xoke's Podcasting Script - Xoke | 2010-05-24

Xoke talks about his podcasting script that is available on xoke.org


bash loops - Ken Fallon | 2010-03-24

user@pc:~$ for number in 1 2 3
> do
> echo my number is $number
> done
my number is 1
my number is 2
my number is 3

user@pc:~$ for number in 1 2 3 ; do echo my number is $number; done
my number is 1
my number is 2
my number is 3

user@pc:~$ cat x.txt|while read line;do echo $line;done
one-long-line-with-no-spaces
one ling line with spaces

user@pc:~$ for line in `cat x.txt`;do echo $line;done
one<-long-line-with-no-spaces
one
ling
line
with
spaces