How I prepare HPR shows (HPR Show 1938)

Dave Morriss


Table of Contents

Introduction

I have been contributing shows to Hacker Public Radio since 2012. In those far off days (!) we sent everything in via FTP, and had to name the files with a combination of our host id, our name, the slot number and the title. The show notes had to contain a chunk of metadata in a defined format to signal all of the various attributes of the show. I found myself making numerous mistakes with this naming and metadata formatting and so started designing and writing some tools to protect myself from my own errors.

I started developing a Bash script in mid-2013 which I called hpr_talk. I used Bash since I thought I might be able to make something with a small footprint that I could share, which might be useful to others. The script grew and grew and became increasingly complex and I found I needed to add other scripts to the toolkit and to resort to Perl and various Perl modules to perform some actions.

Then in 2014 Ken changed the upload procedure to what it is now. This is a much better design and does away with the need to name files in odd ways and add metadata to them. However, this left my toolkit a bit high and dry, so I shelved the plans to release it.

Since then I have been enhancing the hpr_talk toolkit, adding features that I found useful and removing bugs, until the present time. Now it is probably far too complex and idiosyncratic to be of direct use to others, and is rather too personalised to my needs to be easily shared. Nevertheless, it is available on GitLab and I am going to describe it here in case it (or the methods used) might be of interest to anyone.

Overview

As I have already said, the main script is called hpr_talk. It and its supporting scripts and files need to be unpacked into a directory where the shows are going to be stored. The principle is that each show and its related files are stored in their own sub-directory. The script derives the name of the directory from the title of the show it holds.

I keep my scripts and HPR episodes in the directory ~/HPR/Talks/, but there's no problem with placing them anywhere so long as the directory is readable and writable by the account used to manage things, and everything is kept together.

The hpr_talk script takes one or two arguments. The first is an action word which specifies the function you are carrying out (such as install or create). The second argument varies with the function you are using and is optional. If present it can be the name of the directory holding your show or can be the title of a show.

If the argument is -h then the script displays a help message.

My Work-flow

I have tried to give a fairly brief summary about how I use hpr_talk when creating and developing a episode for HPR. These are the main steps:

  1. Create a new episode. I sometimes do this when I have an idea for a show, just as a placeholder. During creation the script collects all the metadata relating to the show by asking questions, setting defaults where I'm not yet sure of the answer.

  2. Having created the show, I might mess around with the title, summary, tags and so forth before I'm happy with them. This can be done at any time in the life-cycle of the episode.

  3. Mostly the shows I create consist of brief notes for the main page, and some longer notes available off that page (this one is an example). My notes are all prepared in Markdown, but I convert them to HTML before uploading. I might have example files, scripts and pictures in some cases. If I do then they and the long notes are packaged into a compressed TAR file for upload.

  4. Since I'm generating HTML from Markdown I have added features to hpr_talk to aid me with this process. I create boilerplate Markdown files with the script, and I generate a Makefile which allows me to automate the build using GNU make. Since I use Vim to edit my notes I generate the configuration file necessary to run the Session plug-in. This way all I have to do is open the session in gVim and all the files are ready to edit.

  5. If I am including multiple files in my show, such as pictures or scripts, I list them in a manifest file. The hpr_talk script allows me to create this file, and the Makefile uses it when generating the TAR file.

  6. As I work on the notes I rebuild the HTML version and view it in a browser. The build process can be run out of Vim using the make command, or through the hpr_talk script on the command line. References to files and images are relative to the local environment at this stage, so the links work, but the command make final (or an appropriate script option) can be used to build everything to work on the HPR server.

  7. The audio is generated independently, perhaps using Audacity or a portable recorder. I save the raw audio in the show directory and edit it and finally export it to the show directory. I use Audacity to save generic audio tags, but I have the means of generating specific tags automatically from the configuration file. This is not currently part of the system available on GitLab due to the number of dependencies.

  8. As the components of a show accumulate the files are "registered" through a script function. The design is such that the primary elements are:
    • the main notes
    • the supplementary files bundled into a TAR file (as defined by the manifest file)
    • the audio
  9. Once everything is ready to be uploaded I reserve a slot and then use hpr_talk to record the selection in the configuration file. This causes files to be renamed automatically to reflect the show they belong to. It also causes any HTML references to be changed in the notes when I perform the final build. I fill in the upload form manually with the details from the configuration, and the main notes. Then I use hpr_talk to perform the upload, which it does using curl to write to the FTP server.

Features of the hpr_talk script

The actions supported by the script are:

Install the script

This is something that is normally done only once. It sets up all of the configuration files needed by the script. It asks for the HPR host id and host name, so is really only appropriate if the user has already received these. This function also allows updating so is useful if something changes like the FTP password, for example.

Create a new show

As mentioned previously, the script will prompt for a number of parameters which it will store away in a show-specific configuration file in the show directory (which it will create). While the eventual show number (slot) is unknown the main files are called hpr____.* and then renamed when a slot has been reserved.

Change the configuration of a show

This function allows viewing and manipulation of the configuration of the show. For example, if you want to change the title, or update the summary, or when you have decided on the slot.

Advanced configuration

Here some of the more complex features can be set up:

  • Create the main show note template
  • Create the full (extended) show note template
  • Create and populate a manifest file (called manifest)
  • Create a Makefile to drive the building of the show files. Several options can be enabled here, such as whether to produce ePub notes
  • Create a Vim session. You need to be using Vim or gVim with the 'Session' plug-in to use this.

Build the notes

As already mentioned this function runs various types of make commands to perform the actions needed to build notes.

  • Build notes: build the main notes into HTML
  • Build all: build all of the options chosen for the show such as full notes, ePub notes, TAR file. Do this for local viewing
  • Build final: same as 'build all' but use URLs suitable for the HPR site
  • Touch all: refresh all files to force make to rebuild them

Register the files you're going to upload

This lets you register the various files you have prepared for the show. It stores their details in the show configuration file.

Release the show

This is for uploading the show via FTP. You need to have registered everything in the configuration file first and should have built the final versions of everything.

Report the status of a show or all pending shows

There are two functions here. The first, status, prints a summary of the state of a show, mainly reporting the contents of the configuration file.

The summary function scans through all of the shows you have prepared but not uploaded and reports on their current state. This is useful if you are like me and have multiple ideas and partially completed shows stored in the main directory.

Example use

The following shows the output generated by hpr_talk when checking the status of this particular episode. The audio is done and registered, but a slot has not yet been chosen:

 cendjm@i7-desktop:~/HPR/Talks$ ./hpr_talk status How_I_prepare_HPR_shows
 Status of an HPR talk
 --------------------------------------------------------------------------------
 
  1: Hostid:    225
  2: Hostname:  Dave Morriss
  3: Email:     perloid@autistici.org
  4: Project:   How_I_prepare_HPR_shows
  5: Title:     How I prepare HPR shows
  6: Slot:      -
  7: Sumadded:  No
  8: Inout:     No
  9: Series:    -
 10: Tags:      Markdown,Pandoc,ePub,Bash,Perl,FTP
 11: Explicit:  Yes
 12: Summary:   I use my own tools for preparing my HPR shows. I talk about them in this episode
 13: Notetype:  HTML
 14: Structure: Flat
 15: Status:    Editing
     Dir:       /home/cendjm/HPR/Talks/How_I_prepare_HPR_shows
     Files:     hpr____.flac
     Files:     hpr____.html
     Files:     hpr____.tbz
 
 Size of HTML notes: 2429 characters

Show Notes

As already mentioned the notes are expected to be in Markdown format. The tool used to process the Markdown is pandoc and certain assumptions have been made about how this is run. For example, the main notes are built as an HTML fragment, while the extended notes are built stand-alone. Also the extended notes refer to the HPR site for their CSS to make them compatible with the HPR look and feel.

If the ePub option is chosen then certain assumptions are made about the layout of the end product. This part of the design is really in a state of flux at the moment and is very much attuned to my tastes.

The notes are really a template

The main and supplementary notes files are both passed through a pre-processor before being read by pandoc. This is a Perl script which interprets expressions in the Template Toolkit syntax. The pre-processor is given arguments consisting of the names of files such as the extended notes and the contents of the manifest file. This simplifies the process of linking to supplementary files and images and allows the generated URLs to change depending on whether the HTML is for local viewing or is the final version for upload.

For example, my main notes file often ends with text such as the following:

I have written out a moderately long set of notes about this subject and these
are available here [[% args.0 %]]([% args.0 %]).

Here [% args.0 %] is an expression which substitutes the first argument to the pre-processor into the text. These expressions are enclosed in a Markdown link in this example.

I also recently prepared a show with multiple images and inserted them thus:

[%- DEFAULT i = 0 -%]
![Preparing carrots 1]([% args.$i %])\
*Slice the carrots diagonally*

![Preparing carrots 2]([% i=i+1; args.$i %])\
*The slices should be moderately thick, about 5mm*

This generates the argument references programmatically, which I found was easier to manage when there were 30+ images.

Conclusion

The toolkit described here does a large amount of what I want when preparing HPR shows, and saves me from my failing memory. The present design still shows the signs of its origin in the days before the current submission mechanism, and this will be corrected in time.

I'd like to automate the completion of the submission form in a later version, though whether Ken will appreciate me attaching scripts to it I doubt.

You are welcome to try the scripts out if you want; that's why it's on GitLab. There is not much documentation at the moment, but I am adding to it gradually. Please contact me if you have problems or suggestions on how to improve the project.