This NHPR episode 2007-1936 entitled, responds to show 2007-1920, and in part of the series,
Bash Crypting.
It is posted by name Boris and in about 18 minutes long, and Karina next visit flag.
The summary is some suggestion on how to improve a Bash Crypt.
This episode of HBR is brought to you by an honest host.com.
At 15% discount on all shared hosting with the offer code HBR-15, that's HPR-15.
Which are web hosting that's honest and fair at an honest host.com.
Hello everybody, this is Dave Morris.
Welcome to Hacker Public Radio.
Today I'm doing a show that I'm not of the type that I've not done before.
I'm responding to show number 2720.
This was a show that Ken Fallen did on January 4th, and he was talking about a script
where it was written to Dan Lode YouTube channels using RSS feeds.
The offer to Bash Crypt called YouTubeRSS.bash, and talked about how it worked and
stuff.
I thought it was very cool.
I've not done this sort of thing myself, but I've sort of filled around in the
particular region of things, so it was quite an interesting lesson I found.
One of the things I do, which is, I don't know if it's good or bad, might be irritating,
is when I see a Bash Crypt, I usually find one must have looking at it with an eye to
rewriting stuff to make it fit better with the stuff that I've been learning, while I've
been doing this series of Bash Tips that I do, and I think it's a useful thing to do
because it's a way of understanding a script.
It's also sometimes people have much better ideas than the ones I was going to offer,
so I can learn from them.
It's useful when it comes to my own scripts.
I've been writing Bash scripts since the 1990s, or when we had Bash on unique systems,
I'll trick, so I think it was.
I'm always looking at my own scripts thinking, that was really, that's not a good way of
doing it.
I could improve on that.
So I do a lot, do this a lot, okay?
So looking at the transcript, that it's needless to say, I saw some areas for improvement.
And I thought that making a show about it would be a way of sharing my thoughts about them.
As we're quite low on shows, as I'm putting this together, it's also motivated me
to do a show rather than just write comments to the kids show or send them an email.
So in general, there are a few uses of walloops in the script that I would be inclined
to rewrite, and I'll probably say a little bit more about them a little bit later, but since
the script works as it stands, then it's more advisory than me saying you really should
change that.
When I did was I grabbed the YouTube RSS script and put it into them.
And there I have a tool called shell check, which is being run by another tool, which
whose purpose is to examine bash scripts.
I've mentioned this before.
And quite a lot of issues were thrown up, most of them were quite trivial, but a few
are a bit more serious.
And in order to give you some sort of taste of what this might mean, or how this would
be represented, I did a screenshot of an editor session, which I've turned into an image
for this show.
There's a thumbnail, which you can click in the long notes, which will take you to a picture
part of the file and the error messages that shell check is producing errors and warnings
really.
So it's worth maybe looking at a few of these, I could spend ages going into this in depth,
but even I'm not that keen to do that, I'm sure you're not keen to listen to it.
First one is on line 34, and shell check doesn't like the expression, open double quotes,
dollar, open curly bracket, log file, closed curly bracket, closed double quotes, dot, dollar,
open parenthesis, slash bin slash date, et cetera, closed parenthesis.
What Ken's doing here is taking the log file variable, and he's concatenating that with
a full stop, a dot, and the result of running the date command with date stamp, year
month, day, a minute second type thing.
Great, I do these sort of thing a lot, it's going to be good, but shell check doesn't
like it because it's working on the principle, but what comes back from date could
have spaces in it, in which case the shell will pass the whole string, and we'll say,
oh, the space isn't there, it might well split it up in whatever context it's being
used.
If you adjust it and put the quotes around the whole expression, then it's perfectly
happy.
Now, think about shell check, I'm going to show on it later anyway, but it will be saying
things, it's not, it's very clever, but it's not, it's clever as a human, and so it will
be reporting things that just look like they could be a problem, but not necessarily.
This is not in itself a problem, personally, I fix these things in my own scripts.
Just to shut the thing up.
Second point, on lines 39 or 48, there are instances of the read command, but it doesn't
use the hyphen R option, hyphen R is recommended because it prevents the, any backslash
character in the input from being mangled, not quite exactly what happens, but anyway,
it's recommendation, and so shell check moons about that.
I found just going and adding them into my copy of it, made it stop.
Also on the same two lines 39 or 48, there are these wild loops, and the wild loops
are of the form, something a rather that generates data, a series of commands or something
piped into, while, and then after that, the body of the loop and done.
Now, I've said this in some of my other shows on the bash tips heading, that if you do
that, it means anything that you change in the loop body, that any variable that you
change in the loop body, don't get passed back to the main script, and that can catch
you out.
So in general, I avoid loops of that structure.
Now, in Kent's case, there's no need to do that, but I'm just saying, and I normally
turn them into a wild, read, et cetera, body of the loop, and then at the end of the
loop, I put a process substitution, which generates the data for the loop, and in that
case, you're not going to have issues with passing variables back to the main script.
Now, it's not a series issue with Kent's script, it's just a thing that's probably
a wise habit to get into of writing loops in a way that they can't catch you out.
As they stand in Kent's script, it's not an issue, but if he were to come back to a later
stage and put up some accumulator in there or something rather happening in there, they
want to use in the main script, he would find it wouldn't work as the thing's structured
now.
So it's got idea just again, I think.
You may think differently, but it's just about some hints and tips that I found on my journey.
So I wanted to look at three instances of things I would be inclined to rewrite.
Again, I've referred to them by line within the script as it was downloaded from Kent's
show 2720.
First off is line 50, and in this line, Kent's doing an if test, and the test is in
square brackets, and it consists of in quotes, the command substitution which uses
grep, and it's comparing a variable called this video with log file.
Log file is a file, it's going to look for the string this video in the actual file.
If it gets anything back, whatever it gets back, will be counted in a pipe with the WC
command, WC minus L, which will count the number lines that come back.
So there could be no lines or there could be one, possibly more.
And this expression is then compared with zero.
So this is very commonly said, I would say, I think I did cover some of these sorts of
things in some of the other shows I've done.
One I would do with this is I would rewrite it, and my rewrite would be to use the
fact that grep will return a true false value, and you can tell it not to return any lines
off of data, so only grep will pull stuff out of the file that matches whatever you're
trying to match.
If there are matches, that's what Kent was trying to do, and he was counting the number
that came back.
So you would do if, and you don't need to put this in square brackets at all, you would
use an exclamation mark to invert the result of the grep test.
And it would be grep space, hyphen Q, that means to that grep is to not producing output
Q for quiet.
Then the variable this video in quotes, and then the log file name.
So if this video is found in the log file, then the not in front of it, the exclamation
mark will turn that to false, if it's not found, then it will return true.
So in other words, it's doing the same thing, it's looking to see if the string in this
video is not in the log file.
I think Kent was trying to determine whether he'd already downloaded something on.
Moving on to line 68, I've got another rather convoluted if statement, and this particular
case is complicated by the fact that the downloadable version of the script has got a variable
called skip crap, which I think is great name, and it's been commented out.
So you're going to find that shell check is going to say, well, there's no variable call
script crap.
How come you're doing using this, and I imagine it's sort of waving its hands in the air
and running around in circles, but that's just me.
I reenable this.
This was Kent's plan to check for certain nonsense in the titles of videos coming back.
So what he has here is in double square brackets, he has a not an exclamation mark,
then he's got hyphen z and then the variable skip crap in quotes and curly brackets,
because sometimes that's a good idea.
So he's checking to see if the skip crap variable is zero length, and then he's reversing
the answer.
So he's actually asking to whether it's non-zero length.
So that's a bit, that's a bit back to front, I reckon.
I replace that with hyphen n and the variable skip crap.
That's then followed by double ampazand, so he's looking for the case where skip crap
is not empty, effectively, because otherwise it wouldn't do any rest of the test.
And the original uses another command substitution dollar open parenthesis, and here
can echoes the title, which will be the title of the current video that is being processed,
into eGrep with the hyphen i option, a using variable skip crap as the string to compare.
So what that's doing is it's telling eGrep to compare the contents of skip crap, which
is a regular expression, it's a bunch of words separated by vertical bars, but it's the type
that eGrep can handle.
And comparing that with what comes in on it's standard input, which is title, which
has been echoed to it by the previous command in the pipeline.
Then if anything comes from everything that comes from back from that is counted with
a WC minus L, and on the completion, the closed parenthesis, the completion of that,
command substitution, there will be a value, which is being compared with zero, it's
a not equal to zero.
So I did a first rewrite, which again echoed title to a changed eGrep, because shell
check was complaining about it.
There is no eGrep command, officially any more eGrep is actually grep space hyphen capital
e.
If you look up the man page for eGrep, you don't find one, it takes you to the grep one.
Then I told it to be quiet with a hyphen Q, then Ken's hyphen I, and then the variable
skip crap.
So I changed that, and I did it outside of the double square brackets, because it's just
a pipeline, which is doing some stuff, and it's returning a true or false value.
So this is a somewhat more tidy.
Well, it's certainly not doing the the thing where you're getting grep to put some stuff,
and then you're counting it, and then you're comparing the result, the number of lines
with a value, because we just want a true or false value.
I've already said some of this, or I will beat myself, and so this actually works.
Then I thought, well, really, what's happening here is the stringing title is being
compared with the stringing in skip crap using a regular expression match, except that
it's using grep, or eGrep, to do it.
So I thought, what would happen if you use batches regular expressions?
So I wrote a version of this with everything in the double square brackets that you have
to do for regular expressions, and replaced the the pipeline with dollar title equals
tilde dollar skip crap.
I'm not a great one for the curly brackets, or I don't need them, but it's
up to you how you do that.
So I did a little test.
I copied Ken's skip crap string into skip crap on the command line, created a title,
the best bets in the world, and then I ran the little little test against the
test string, the regular expression test.
If it matched, it was to echo crap, otherwise it echoed non-crap.
And then Ken's looking for best pet stuff, then it worked.
The downside with that answer is that the test is case sensitive, whereas with grep,
the hyphen i made it case insensitive, but there's a way around that, and the way he is to
make sure that both title and skip crap are forced to lower case just for the test.
So I gave you an example of how you would do that, and that works fine.
If I changed the case of the title and my experiment, I didn't put this in the notes.
Then it also matches, it matches whatever the case is, as that's what I'm saying to do.
The last one, line 88, Ken has a massively long line, which starts with cat, and then in double
quotes, dollar log file underscore to do.
This is the case where he doesn't need the curly brackets, piped into the YouTube DL command
with all of its various arguments.
As shell check complains about this, and says it's what it calls a useless cat,
and the full message are copied into the notes here.
Says useless cat consider command less than file, piped into something rather or,
command file, piped instead.
So basically it's saying that you can probably replace,
where you should be replacing that cat, because all cat is doing is taking a file
squirting it out on standard out.
So it's caught by the thing down the pipe on its standard input channel,
and then processed.
Now I don't know YouTube DL that well.
I would be surprised if you can't give it a file as its argument.
But in the assumption that you can't, and it's expecting stuff on its input channel,
when you can write the string using YouTube DL to start with,
and with all its arguments, and then at the end of it,
use redirection less than sign and the name of the file that you're going to be reading from.
So that would do it.
The other way you can do it, and it's perfectly legal, but it's the thing I personally never
do, is to start the whole line with the less than the name of the file,
and then YouTube DL, et cetera, et cetera, et cetera.
There's Wikipedia article on the use of cat,
and there's also a reference to the page UUOC,
which stands for useless use of cat.
Wait, there was a thing that was some sort of competition for an award for the most
useless use of the cat command in the Unix.
I'm not sure it's still alive, it's referenced in the Wikipedia article.
The process of removing useless cats is called demorgification,
which I'm told is a Britishism, because we tend to call cats moggies,
and I don't know if that's specifically British or what, but that just
it means me.
So, in conclusion, Bash is powerful, but does them things,
and there's just a skill way we know this very well, shell checks, helpful tool,
helps you catch scripting errors, but it can be a little bit of an
irritant when it nags about things.
You can shut it up in various ways.
I'm preparing another HPR episode about it's use with them,
and the use of it with other checkers.
It's really powerful.
So I thought people would be mildly interested in that.
Finally, apologies to can, if it seemed I was making
excessive criticisms of his script, it was meant to be
constructive criticism, of course.
And as I said at the start, Ken was asking for constructive criticism.
So, there we go.
Anyway, I hope you've found that useful, and, uh, catch you next time.
Okay, bye.
You've been listening to Hecker Public Radio at HeckerPublicRadio.org.
We are a community podcast network that releases shows every weekday
Monday through Friday.
Today's show, like all our shows, was contributed by a HPR listener like yourself.
If you ever thought of recording a podcast,
then click on our contributing to find out how easy it really is.
HeckerPublicRadio was found by the digital.com and the InfoNomicon computer club
and is part of the binary revolution at bmf.com.
If you have comments on today's show,
please email the host directly, leave a comment on the website, or record a follow-up
episode yourself.
On this otherwise status, today's show is released on the creative comments,
a description, share a like, follow-up video, and follow-up video.