Site Map - skip to main content

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.


hpr3413 :: Bash snippet - using coproc with SQLite

Sending multiple queries to a running instance of sqlite3

<< First, < Previous, Latest >>

Host Image
Hosted by Dave Morriss on 2021-09-01 is flagged as Explicit and is released under a CC-BY-SA license.
Tags: Bash,coproc,subshell,coprocess,pipe,file descriptor.
Listen in ogg, spx, or mp3 format. | Comments (4)

Part of the series: Bash Scripting

This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.

Introduction

I am in the process of rewriting some scripts I use to manage Magnatune albums. I’m a lifetime Magnatune member and have access to the whole music collection. I wrote a script for downloading albums and placing them in my ~/Music directory which I talked about in 2013 (show 1204). The original scripts are still available on GitLab and I know of one other person who made use of them!

Since 2013 I have written a few other support scripts, for example one to manage a queue of albums I want to buy and download, and one which summarises the state of this queue.

It’s this 'show_queue' script I am currently updating (called show_queue_orig, and available in the resources to this show). The original version of this script took Magnatune album URLs from a file (acting as a queue of stuff I wanted to buy), parsed out a piece of the URL and used it to grep a pre-prepared summary in another file. This file of summaries had been made from a master XML file provided by Magnatune (see update_albums on GitLab).

Magnatune has moved away from this master XML file to a SQLite database in recent years, so I want to perform a database lookup for each URL to list its details.

The first version of the new script wasn’t difficult to write: just extract the search data as before and run a query on the database using this data. I have included this script which I call show_queue_db_1 amongst the resources for this episode, so you can see what I’m talking about – and what I want to improve on. It felt bad to be performing multiple calls on the sqlite3 command in a loop, so I looked around for an alternative way.

In April 2019 clacke did a show (number 2793) about the Bash coproc command.

This command creates a subshell running a command or group of commands which is connected to the calling (parent) process through two file descriptors (FDs). It’s possible for the calling shell to write to the input descriptor and read from the output one and thereby communicate with whatever is running in the subshell.

I was vaguely aware of coproc at the time of clacke’s show but hadn’t looked into it. I found the show fascinating but didn’t have a use for the feature at the time.

To solve my need to show my Magnatune queue of future purchases, it looked as if a sqlite3 instance running in a subshell could be given queries one after the other and return the answers I needed. My journey to a Bash script using coproc then followed.

Long notes

Follow this link to read the detailed notes associated with this episode.


Comments

Subscribe to the comments RSS feed.

Comment #1 posted on 2021-09-01T01:14:43Z by b-yeezi

New tool for my toolbox

Thank you, Dave, for this great show. I will be definitely using coproc in the near future.

Comment #2 posted on 2021-09-02T00:47:24Z by Trey

Excellent detail!

Thank you for the detailed explanations in this episode. I was completely unfamiliar with coproc before listening. Now I have something new to play with, and am learning something new.

Looking forward to the next bash episode.

Comment #3 posted on 2021-09-02T11:51:13Z by Dave Morriss

Re: New tool for my toolbox

Thanks b-yeezi,

Glad you enjoyed the show. I hope you find coproc a useful thing. An HPR show on your experiences would be very welcome I'm sure!

Dave

Comment #4 posted on 2021-09-02T20:07:03Z by Dave Morriss

Re: Excellent detail!

Thanks Trey,

Glad you found the show useful. I sometimes wonder if I'm overdoing the detail, but I enjoy getting into the intricacies of stuff and like to share what I find.

I'm planning Bash Tips episode 22 at the moment, so it should be out before too long.

Dave

<< First, < Previous, Latest >>

Leave Comment

Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.

Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).

Provide feedback
Your Name/Handle:
Title:
Comment:
Anti Spam Question: What does the P in HPR stand for ?