May 06

Perl rename

http://www.perlmonks.org/?node_id=303814

This comes by default with debian derivs, too bad there’s not a clean download for redhat somewhere…

Feb 22

Checking rsync or cp backups with md5sum

It’s nerve-wracking to move gigabytes of data around and not be absolutely sure that everything is there completely. A neat (but time consuming) way to do a check without creating huge intermediate files is to generate md5 checksums for each file on the local side, then send the results to the remote side and check those md5 hashes against the corresponding remote checksum.

# The full command:

$ find ./Ian -type f -print0 | xargs -0 md5sum | ssh www 'cd /home/owncloud/data/mcgowan/files ; md5sum --quiet -c -'

Let’s unpack that into the 3 components.  It can be helpful to run each of these commands in isolation to see what they produce.

1. find ./Ian -type f -print0

This command searches the ./Ian directory for all files (-type f, no directories or other things), and then outputs the list of files to stdout delimited with null characters (-print0).  The -print0 is useful because the only characters that you can absolutely guarantee not to be in a filename are / and null.

2. xargs -0 md5sum

This is the easy one.  For each null-terminated (-0) parameter passed from the find command, run the md5sum command and output the checksum and filename to stdout.  This looks something like “d41d8cd98f00b204e9800998ecf8427e  ./Ian/null.txt”

3. ssh www 'cd /home/owncloud/data/mcgowan/files ; md5sum --quiet -c -'

There’s a lot going on in this command.  We’re using ssh to run a command on another server (www), not login to it.  A subtle point here is that the stdout from the previous command (md5sum) is connected to the stdin of whatever command is run on the remote side. This is extending our pipeline of commands to a different machine (in this example, a machine in a different country!).  Another easy-to-miss trick is that we are not getting prompted to login to www, because we are using ssh-agent and password-less ssh logins.

The command run by ssh is everything between single quotes.  Since the files are in a different directory on the remote side, the first sub-command is a “cd” to get us to the right place.  Then we run “md5sum -c” to check the lines of data that are piped into our ssh command.  Since we don’t want to see files that match, the “–quiet” option is added.  Remove that when starting out to see a lot of “OK” messages that make you feel good about your pristine backup.

Dec 26

Wishlist 2014

Continuing with the life-recorder idea, I want a wristband (about the same size as a fitbit) that continuously records background audio, in a loop.  Fitbits When something interesting happens, press a button to save the last 5 minutes.  Kind of like the “heard” app, heardapp.com (“cute things your kids say, spousal disputes”) but somehow seems less creepy when it’s out in the open on your wrist…

Dec 26

Car black box/driving gamification

icarSo, like all red-blooded men, I’m absolutely convinced my driving is better than my wifes (wives? wife’s?  wifes’?  Where does the possessive go there?).  And she is equally convinced of the opposite.  What I really want is an app that monitors our driving (particularly average distance to cars in front), records in a loop and automatically grades a particular driver.  I can see how the constant feedback would improve driving behavior.  Plus I would pay quite a bit more than $0.99 to be proved right…

There are a couple of apps that are close, but not quite there:

Dec 23

Monitoring – back to the stone age

Need some ultra-basic monitoring, and icinga/nagios turned into multiple hours of headache.  Gave up and went back to an old favorite – BigBrother.  The forked version is now called xymon (I bet the Hobbit movie producers didn’t like the original name, however cute it was). Easy to install:

http://xymon.sourceforge.net/xymon/help/install.html

But a frustrating exercise with Debian/Apache – there’s a tussle about where config files go.  The debian packages load them to conf.d, but apache seems to ignore that directory completely.  The upcoming apache release hints at the solution – packages that need to include apache conf instructions are moving to conf-available. https://wiki.debian.org/Apache/PackagingFor24

#cp conf.d/xymon ../conf-available/xymon.conf
#a2enconf xymon.conf
#service apache2 reload

Then there’s the old permissions language that needs fixing.  @DaveRandom on this stack overflow thread is an internet saint!

http://stackoverflow.com/questions/11992466/apache-permission-denied

Nov 28

Migrating to owncloud on dedicated server

ovhSo, the plan is to get combine several cloud services – dropbox, azure (windows remote desktop for swim team), backupsy, linode – onto one $29/mo box at ovh.com, http://www.ovh.com/us/dedicated-servers/pe1.xml

After installing, which was super quick and easy, going thru adding the usual stuff:

ufw
automatic updates
etckeeper
ssh keys only
fail2ban
apache + mysql
mediawiki
wordpress
virtualbox headless

What else are fun/useful things to have on a shared host?

[update 12/16/2013 - part way there.  Azure, Backupsy and Linode cancelled now.  Dropbox next, after installing owncloud desktop sync]