skip to main content - accessibility menu - dyslexic font - mobile - text - print

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.

hpr1430 :: download script

A hacked script to download youtube videos

<< First, < Previous, Latest >>

Hosted by Ken Fallon on 2014-01-24 and released under a CC-BY-SA license.
Listen in ogg, spx, or mp3 format. | Comments (8)

Part of the series: Bash Scripting

This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.

In episode "Thu 2013-12-19: hpr1404 Editing pre-recorded audio in Audacity" I walked you through editing a podcast, by the magic of editing this is been posted after the other show has aired. The plan here is to get people to share their useful hacks to show how elegant, or in my case ugly, code can be. As Knightwise says "Getting technology to work for you."™
Feel free to share your own hacks with us.

# Downloads videos from youtube based on selection from
# (c) Ken Fallon
# Released under the CC-0

savedir="${savepath}/$(\date -u +%Y-%m-%d_%H-%M-%SZ_%A)"
mkdir -p ${savedir}

# Gather the list
seq 1 ${maxtodownload} | while read videopage;
  thisvideolist=$(wget --quiet "${videopage}" -O - | 
  grep '' | 
  sed 's#^.*' | 
  awk -F '"|?' '{print ""$1}')
  for thisvideo in $(echo $thisvideolist);
    if [ "$( grep "${thisvideo}" "${logfile}" | wc -l )" -eq 0 ];
      echo "Found the new video ${thisvideo}"
      echo ${thisvideo} >> ${logfile}_todo
      echo "Already downloaded ${thisvideo}"

# Download the list
if [ -e ${logfile}_todo ];
  tac ${logfile}_todo | youtube-dl --batch-file - --ignore-errors --no-mtime --restrict-filenames \
    --max-quality --format mp4 --write-auto-sub -o ${savedir}'/%(autonumber)s-%(title)s-%(id)s.%(ext)s'
  cat ${logfile}_todo >> ${logfile}
  rm ${logfile}_todo


Subscribe to the comments RSS feed.

Comment #1 posted on 2014-02-04T15:27:52Z by Roan

seq will do descending counts

seq 100 -1 1


Comment #2 posted on 2014-02-08T17:22:17Z by Cloud

Great podcast and brilliant idea for a series, but...

now I need to upgrade my broadband to allow for all these great videos that I wasn't getting before!

Comment #3 posted on 2014-02-09T11:14:28Z by Dave Morriss

The power of modern Bash

The power of modern Bash
I wondered why you used:

seq 1 ${maxtodownload} | while read videopage;

as opposed to:

for (( videopage=1; videopage<=${maxtodownload}; videopage++ ))

or (if you don't like repeating 'videopage' three times):

for videopage in {1..10}

You can even do more fancy stuff like:

for i in {001..0010}; do

for i in {0010..001}; do

for c in {a..h}; do

I find I almost never use the 'seq' command in today's version of Bash.

Comment #4 posted on 2014-02-16T10:51:39Z by Ken Fallon

Because ...

I got into the habit of using while loops because it deals with spaces in input better or so I've found., but mostly I can work in "blocks" up to the pipe "|" is one block. Test. Debug. Then on to the next block. That makes It easier to debug on the command line, where most of these start.

Not using seq makes the script too bashey :) but that argument holds little water I know.

Comment #5 posted on 2014-05-23T15:07:11Z by Jim Zatorski

Extra video downloaded

Is anyone else having an extra video appear EVERYDAY (usually the same one)?

I have tracked it down to the "--max-quality" switch. The man page shows an expected "=FMT" clause.

Comment #6 posted on 2014-06-10T08:10:42Z by APCR

Thanks for the script. It works great. Can anyone tell me how to run it as a cron job? I have copied the file to /etc/cron.daily but it does not run. Do I have to run a script that actions this script?

Comment #7 posted on 2014-06-10T14:34:22Z by Ken Fallon


To get this to work in cron you need to make sure that the script is executable. Assuming the script is called "boyt.bash" and is in your own bin directory "/home/apcr/bin/boyt.bash".

chmod +x /home/apcr/bin/boyt.bash

Check that is runs by just typing:

After that it should run in cron.

Comment #8 posted on 2015-03-03T04:59:39Z by Ian

I have copied the script but when I try to run it it says:

toshy@toshy-Satellite-A300:~/Desktop$ ./
awk: line 0: regular expression compile failed (missing operand)

Comment #9 posted on 2015-03-05T09:29:31Z by Ken Fallon

One thing I missed is that the logfile needs to exist the first time you run it so it may produce errors.

@Ian I just tried it on another computer and it didn't complain. It could be that copying and pasting from the web page is causing problems. Try downloading it with wget

wget -O ./

then running it

sh +x ./

Leave Comment

Powered by Comment Script