Site Map - skip to main content - dyslexic font - mobile - text - print

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.


Correspondent

Tony Pelaez

Host Image
Host ID: 263

I feel information should be free, so I created http://ilearnthings.com


email: tony.nospam@nospam.pelaez.me
episodes: 2

hpr1791 :: Organizing Photos with Bash

Released on 2015-06-15 under a CC-BY-SA license.

Summary

In this episode I provide an overview of how I use bash to automate my process for orgainizing photographs on my computer.

There are two main objectives of this script:

  1. Organize photographs in a folder structure that makes sense to me, e.g. 2015/2015-05-22
  2. Allow me to back up my photographs using a variety of methods.

Download the Script

This script is hosted on Github and you can download the latest version using following command:

git clone https://gist.github.com/81e489b2a7397bb17305.git

Script


#!/bin/bash

shopt -s -o nounset

# Create variables and configure script.
declare -rx SCRIPT=${0##*/}
declare TMPDIR=/tmp/photos
declare -r CURRENTDIR=`pwd`
declare FILES=$TMPDIR/*
declare DESTINATION=/media/Tyr/Pictures/Photos
declare -r GOOGLEUSER="tnyplz@gmail.com"
declare -r  OPTSTRING="-h, -d:"
declare -r  LONGOPTSTRING="help, destination-directory, no-google-backup, sd-card, tmp-dir, no-delete, backup"
declare RESULT
declare GOOGLE_BACKUP=true
declare SD=false
declare SDDIR
declare NODELETE=false
declare S3=false

# Executable dependencies
declare -rx find="/usr/bin/find"
declare -rx gphoto2="/usr/bin/gphoto2"
declare -rx google="/usr/bin/google"
declare -rx dcraw="/usr/bin/dcraw"
declare -rx rsync="/usr/bin/rsync"
declare -rx rename="/usr/bin/rename"
declare -rx tar="/usr/bin/tar"
declare -rx s3cmd="/usr/bin/s3cmd"

# Sanity Checks
if test -z $BASH; then
    printf "$SCRIPT:$LINENO: please run this script with the BASH shell\n" >&2
    exit 192
fi
# check for find
if test ! -x $find; then
    printf "$SCRIPT:$LINENO: the $find command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for gphoto2
if test ! -x $gphoto2; then
    printf "$SCRIPT:$LINENO: the $gphoto2 command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for google
if test ! -x $google; then
    printf "$SCRIPT:$LINENO: the $google command is not available -- \
aborting\n" >&2
fi
# check for dcraw
if test ! -x $dcraw; then
    printf "$SCRIPT:$LINENO: the $dcraw command is not available -- \
aborting\n" >&2
fi
# check for rename
if test ! -x $rename; then
    printf "$SCRIPT:$LINENO: the $rename command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for rsync
if test ! -x $rsync; then
    printf "$SCRIPT:$LINENO: the $rsync command is not available -- \
aborting\n" >&2
fi
# check for tar
if test ! -x $tar; then
    printf "$SCRIPT:$LINENO: the $tar command is not available -- \
aborting\n" >&2
fi
# check for glacier-cmd
if test ! -x $s3cmd; then
    printf "$SCRIPT:$LINENO: the $s3cmd command is not available -- \
aborting\n" >&2
fi


# Check for Options
# =================

getopt -T
if [ $? -ne 4 ]; then
    printf "$SCRIPT:$LINENO: %s\n" "getopt is in compatibility mode" >&2
    exit 192
fi

RESULT=$(getopt --name "$SCRIPT" --options "$OPTSTRING" --longoptions "$LONGOPTSTRING" -- "$@")
if [ $? -gt 0 ]; then
    exit 192
fi

eval set -- "$RESULT"

while [ $# -gt 0 ]; do
    case "$1" in
    -h | --help) # show help
        printf "%s\n" "
This script helps you automate the process of downloading photos from
your camera, uploading backups to Google Picasa, and syncing the files
with a specified directory.

Dependendies:
  gphoto2
  dcraw
  googlecl
  rsync
  s3cmd

usage: $SCRIPT [options]

Options:
  -h | --help                        Show help for $SCRIPT
  --destination-directory {LOCATION} Set the location where the photos will be
                                     copied to.
  --tmp-dir {LOCATION}               Set the temporary directory where images
                                     will be downloaded to initially. The
                                     default is /tmp/photos.
  --no-google-backup                 Disable uploading low rez copies to Google
                                     Plus.
  --sd-card {LOCATION}               Set the location of the sd card.
  --no-delete                        Do not delete from temp file.
  --backup {FOLDER} {S3 BUCKET}      Create archive from folder and upload to S3.
  "
        exit 0
        ;;
    --destination-directory ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for destination. No destination given." >&2
            exit 192
        fi
        DESTINATION="$1"
        ;;
    --tmp-dir ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for tmp-dir.  No temporary directory given." >&2
            exit 192
        fi
        TMPDIR="$1"
        FILES=$TMPDIR/*
        ;;
    --no-google-backup ) shift
        GOOGLE_BACKUP=false
        ;;
    --sd-card ) shift
        SD=true
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for sd directory. No sd card directory given." >&2
            exit 192
        fi
        SDDIR="$1"
        ;;
    --no-delete ) shift
	NODELETE=true
	;;
    --backup ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for AWS Glacier Backup. Backup folder and vault must be specified."
        fi
        S3=true
        BACKUP_FOLDER="$1"
        BUCKET="$2"
        ;;
    esac
    shift
done

# Functions
# =========

# function to convert a raw image to jpg.
# input: requires the user to specify the file extention ($1).
function convert_to_jpg () {
  FILES2CONVERT=$TMPDIR/*"$1"
  for FILE in $FILES2CONVERT
  do
    FILE2BACKUP=$TMPDIR/Backup/`basename "$FILE" "$1"`'.jpg'
    if [ -e $FILE2BACKUP ]; then
        printf "$SCRIPT:$LINENO: Skipping $FILE, jpg file already exists\n"
    elif [ -e $FILE ]; then
        printf "$SCRIPT:$LINENO: Converting $FILE to $FILE2BACKUP\n"
        $dcraw -cvz -w -o 1 -q 3 "$FILE" | cjpeg -quality 80 -optimize > "$FILE2BACKUP"
    else
        printf "Did not convert $FILE\n"
    fi
  done
}

# function to resize jpeg to upload to picasa
function resize_to_thumb () {
    FILES2RESIZE=$TMPDIR/Backup/* # TODO pass this in as argument along with destination directory
    for FILE in $FILES2RESIZE
    do
        printf "$SCRIPT:$LINENO: Creating thumbnail for $FILE..."
        convert $FILE -resize 2048x2048 $TMPDIR/Backup/Upload/`basename "$FILE" ".jpg"`'_thumb.jpg'
        printf "done\n"
    done
}

# function to import photos
function import_photos () {
    printf "$SCRIPT:$LINENO: Importing Photos\n"
    if $SD; then
        cp -p "$SDDIR"/* .
    else
        $gphoto2 --quiet --get-all-files
    fi
}

# function to remove spaces in file names
function remove_spaces () {
    $find $1 -depth -name "* *" -execdir $rename 's/ /_/g' "{}" \;
}

# function to sort images into direcotries based on date.
# input: directory to sort ($1)
#        directory to sort into ($2)
function sort_images () {
    SORTDIR=$2'/Sorted/'
    for FILE in $1
    do
        printf "$SCRIPT:$LINENO: Sorting $FILE\n"
        DATEDIR=$SORTDIR`date -r "$FILE" +%Y`'/'`date -r "$FILE" +%Y-%m-%d`
        mkdir -p $DATEDIR
        cp "$FILE" $DATEDIR/
    done
}

# function create archive and upload to AWS S3
# input: directory to create an archive for ($1)
#        s3 bucket name ($2)
function archive_folder () {
    ARCHIVE=$TMPDIR/$(basename $1).tar.bz2
    printf "$SCRIPT:$LINENO: archiving $ARCHIVE\n"
    $tar -cvjf $ARCHIVE $1
    $s3cmd put $ARCHIVE $2
}

# Create temporary directory
mkdir -p $TMPDIR
cd $TMPDIR

# Create AWS Glacier archive
if $S3; then
    archive_folder $BACKUP_FOLDER $BUCKET
    cd $CURRENTDIR
    if [ $NODELETE = false ]; then
        rm -rf $TMPDIR
    fi
    exit 0
fi

# Import files from camera
import_photos
printf "$SCRIPT:$LINENO: Importing Photos Done!\n"

# Remove Spaces in Filenames
remove_spaces $TMPDIR

#Convert all files to lower case
printf "$SCRIPT:$LINENO: Converting Photos to Lower Case.\n"
for FILE in *
do
    f=`echo $FILE | tr '[:upper:]' '[:lower:]'`
    mv "$FILE" "$f"
done
printf "$SCRIPT:$LINENO: Converting Photos to Lower Case Done!\n"

# Sort files
sort_images "$FILES" "$TMPDIR"
printf "$SCRIPT:$LINENO: Sorting Images Done!\n"

# Create backup jpgs and upload them to Picassa
mkdir -p $TMPDIR/Backup
cp $TMPDIR/*.jpg $TMPDIR/Backup/

convert_to_jpg ".nef"
convert_to_jpg ".nrw"
mkdir -p $TMPDIR/Backup/Upload
resize_to_thumb

if $GOOGLE_BACKUP; then
    # Upload jpgs to Picassa
    # Requires that you authorize googlecl through the web browser.
    $google picasa create --user $GOOGLEUSER --title "Backup "`date +%Y-%m` $TMPDIR/Backup/Upload/*
fi

# Copy files to final locations
$rsync -ravv $TMPDIR/Sorted/ $DESTINATION # TODO test to make sure destination works

cd $CURRENTDIR

# Remove temp folder
if [ $NODELETE = false ]; then
    rm -rf $TMPDIR
fi

printf "$SCRIPT:$LINENO: Processing Complete!\n"
exit 0


hpr1370 :: Blogging With Octopress

Released on 2013-11-01 under a CC-BY-SA license.

Blogging with Octopress

Static html site generators automate many of the tedious steps that are necessary to create website. Octopress is a static html generator that automates many of the tedious tasks of static html site generators, and comes with a number of reasonable presets, configured right out of the box.

Static HTML Site Generators I looked at:

I settled on octopress for the following reasons:

SASS
Sass adds additional functionality to css such as variables, mixins, scopes, and was a tool that I had previously worked with.
Twitter Bootstrap
Twitter bootstrap is a set of templates that produce nice looking pages that are standards compliant, and adaptive so that they look good at any screen resolution.
HTML5 Video Plugin
I ended up creating my own, but Octopress has a HTML5 video plugin. Unfortunately this only supported H264 video, so I created my own to serve H264, Webm, and Ogv.
Deployment scripts
Octopress comes with rsync, and github pages support out of the box, so you can deploy your site with very little effort.

Requirements:

  • Ruby 1.9.3 or above
  • Git
  • HTML knowledge
  • Text Editor & Terminal

Install Requirements:

In Ubuntu 12.04 I did the following:

sudo apt-get install emacs git zlib1g-dev openssl libopenssl-ruby1.9.1 \
libssl-dev libruby1.9.1 libreadline-dev

Install ruby through rbenv

rbenv (http://rbenv.org)

git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
# set environment in ~/.bash_profile.  Change this to ~/.zshrc if using zshell
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
source ~/.bash_profile # You can change to .zshrc or .bashrc

Install ruby-build to make installing ruby easy

git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build

Install ruby

rbenv install 1.9.3-p194
rbenv rehash

Octopress

http://octopress.org

Install Octopress

git clone git://github.com/imathis/octopress.git octopress
cd octopress
rbenv local 1.9.3-p194  

Install Ruby Requirements

gem install bundler
rbenv rehash
bundle install  

What is rake?

Rake is like make but for ruby.

Use rake scripts to setup and preview blog

rake -T # list all available rake tasks
rake install # install themes and default config
rake preview # generate and view site

Open localhost:4000 in your webbrowser

Setup Deployment

rake set_root_dir['blog-test']
rake setup_github_pages 
rake generate
# Change the following url to point to your repository
git remote add origin https://github.com/HarryGuerilla/blog-test.git
git config branch.master.remote origin
git add .
git commit -m "initial commit"
git push origin master
rake deploy # this is where the magic happens

Configure Blog

emacs _config.yml
# Edit title, author, subtitle

Create First Post & Basic workflow:

rake new_post
emacs post
git add .
git commit -m "added new post"

Publish Blog

rake deploy

Become a Correspondent