Using cron, du,sqlite, and bash to find directory growth
Hosted by norrist on 2016-09-07 is flagged as Clean and is released under a CC-BY-SA license.
Listen in ogg,
mp3 format. | Comments (6)
0 3 * * 0 /bin/du -m /data/ > /home/USER/du_files/"du_$(/bin/date +\%Y\%m\%d)"
YESTERDAYS_FILE="du_$(/usr/bin/date --date="7 days ago" +%Y%m%d)"
/usr/bin/echo "create table old (oldsize integer, path varchar);" > delta.sql
/usr/bin/echo "create table new (newsize integer, path varchar);" >> delta.sql
/usr/bin/echo '.separator "\t" ' >> delta.sql
/usr/bin/echo ".import $TODAYS_FILE new" >> delta.sql
/usr/bin/echo ".import $YESTERDAYS_FILE old" >> delta.sql
/usr/bin/echo ".mode csv" >> delta.sql
/usr/bin/echo ".headers on" >> delta.sql
/usr/bin/echo ".out deltas.csv" >> delta.sql
/usr/bin/echo "select *,newsize-oldsize as delta_in_megabytes from old natural join new where oldsize<newsize order by delta_in_megabytes desc;" >> delta.sql
/usr/bin/sqlite3 < delta.sql
echo $YESTERDAYS_FILE|/usr/bin/mailx -a deltas.csv -s deltas.csv email@example.com
create table old (oldsize integer, path varchar);
create table new (newsize integer, path varchar);
.import du_20160821 new
.import du_20160814 old
select *,newsize-oldsize as delta_in_megabytes
from old natural join new where oldsize<newsize
order by delta_in_megabytes desc;
Comment #1 posted on 2016-09-07T02:35:07Z by mackrackit
I found this episode very useful. It gave me a lot of ideas. Looking forward to more like it.
Comment #2 posted on 2016-09-08T11:34:46Z by JONATHAN KULP
I thought I heard you say at the end of this episode that it might be the worst HPR ever? No way! I really enjoyed this, was great hearing how you worked your way through the problem and arrived at a usable solution. Please do more!
Comment #3 posted on 2016-09-08T16:25:37Z by Gumnos
Cleaning up the script
You could clean up the script a bit by using a "here document" instead of a temporary SQL file, something like
Comment #4 posted on 2016-09-08T19:18:21Z by Kevin O'Brien
As I was listening to this show on my drive in to work I was thinking that it epitomizes what we mean by something of interest to hackers. I want to hear more from norrist.
Comment #5 posted on 2016-09-08T19:28:17Z by norrist
I have never heard of a "here document". Thanks for the tip.
Comment #6 posted on 2016-09-10T22:02:47Z by Dave Morriss
I enjoyed this
A good topic for a show I thought. I enjoyed following your thinking and your solution. The audio was good and the background noise was not distracting at all.
I never use full paths to commands, though I can see cases where perhaps I should. Have you been bitten by not doing this in the past? If so I'd like to hear about it.
I wondered why the date program used in the crontab entry was /bin/date whereas it was /usr/bin/date in the main script. Are you working across different OSes or architectures?
Looking forward to more!
<< First, < Previous, Next >, Latest >>
Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.
Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).