skip to main content - dyslexic font - mobile - text - print

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.

hpr1430 :: thebestofyoutube.com download script

A hacked script to download youtube videos

<< First, < Previous, Latest >>


Hosted by Ken Fallon on 2014-01-24 and released under a CC-BY-SA license.
Listen in ogg, spx, or mp3 format. | Comments (6)

In episode "Thu 2013-12-19: hpr1404 Editing pre-recorded audio in Audacity" I walked you through editing a podcast, by the magic of editing this is been posted after the other show has aired. The plan here is to get people to share their useful hacks to show how elegant, or in my case ugly, code can be. As Knightwise says "Getting technology to work for you."™
Feel free to share your own hacks with us.

http://hackerpublicradio.org/eps.php?id=1404
http://hackerpublicradio.org/eps/hpr1430-downloader.bash.txt


#!/bin/bash
# Downloads videos from youtube based on selection from http://thebestofyoutube.com
# (c) Ken Fallon http://kenfallon.com
# Released under the CC-0

maxtodownload=10
savepath="/mnt/media/Videos/tv/youtube/bestofyoutube"
savedir="${savepath}/$(\date -u +%Y-%m-%d_%H-%M-%SZ_%A)"
mkdir -p ${savedir}
logfile="${savepath}/downloaded.log"

# Gather the list
seq 1 ${maxtodownload} | while read videopage;
do 
  thisvideolist=$(wget --quiet "http://bestofyoutube.com/index.php?page=${videopage}" -O - | 
  grep 'www.youtube.com/embed/' | 
  sed 's#^.*www.youtube.com/embed/##' | 
  awk -F '"|?' '{print "http://www.youtube.com/watch?v="$1}')
  for thisvideo in $(echo $thisvideolist);
  do 
    if [ "$( grep "${thisvideo}" "${logfile}" | wc -l )" -eq 0 ];
    then
      echo "Found the new video ${thisvideo}"
      echo ${thisvideo} >> ${logfile}_todo
    else
      echo "Already downloaded ${thisvideo}"
    fi
  done
done

# Download the list
if [ -e ${logfile}_todo ];
then
  tac ${logfile}_todo | youtube-dl --batch-file - --ignore-errors --no-mtime --restrict-filenames \
    --max-quality --format mp4 --write-auto-sub -o ${savedir}'/%(autonumber)s-%(title)s-%(id)s.%(ext)s'
  cat ${logfile}_todo >> ${logfile}
  rm ${logfile}_todo
fi


Comments

Subscribe to the comments RSS feed.

Comment #1 posted on 2014-02-04T15:27:52Z by Roan

seq will do descending counts

seq 100 -1 1

seq FIRST INCREMENT LAST

Comment #2 posted on 2014-02-08T17:22:17Z by Cloud

Great podcast and brilliant idea for a series, but...

now I need to upgrade my broadband to allow for all these great videos that I wasn't getting before!

Comment #3 posted on 2014-02-09T11:14:28Z by Dave Morriss

The power of modern Bash

The power of modern Bash
I wondered why you used:

seq 1 ${maxtodownload} | while read videopage;
do

as opposed to:

for (( videopage=1; videopage<=${maxtodownload}; videopage++ ))
do

or (if you don't like repeating 'videopage' three times):

for videopage in {1..10}
do

You can even do more fancy stuff like:

for i in {001..0010}; do

for i in {0010..001}; do

for c in {a..h}; do

I find I almost never use the 'seq' command in today's version of Bash.

Comment #4 posted on 2014-02-16T10:51:39Z by Ken Fallon

Because ...

I got into the habit of using while loops because it deals with spaces in input better or so I've found., but mostly I can work in "blocks" up to the pipe "|" is one block. Test. Debug. Then on to the next block. That makes It easier to debug on the command line, where most of these start.

Not using seq makes the script too bashey :) but that argument holds little water I know.

Comment #5 posted on 2014-05-23T15:07:11Z by Jim Zatorski

Extra video downloaded

Is anyone else having an extra video appear EVERYDAY (usually the same one)?

I have tracked it down to the "--max-quality" switch. The man page shows an expected "=FMT" clause.

Comment #6 posted on 2014-06-10T08:10:42Z by APCR

Thanks for the script. It works great. Can anyone tell me how to run it as a cron job? I have copied the file to /etc/cron.daily but it does not run. Do I have to run a script that actions this script?

Comment #7 posted on 2014-06-10T14:34:22Z by Ken Fallon

cron

To get this to work in cron you need to make sure that the script is executable. Assuming the script is called "boyt.bash" and is in your own bin directory "/home/apcr/bin/boyt.bash".

chmod +x /home/apcr/bin/boyt.bash

Check that is runs by just typing:
/home/apcr/bin/boyt.bash

After that it should run in cron.

Leave Comment

Name
E-mail (Will not appear online)
Website
Title
Comment
Powered by Comment Script