20 June 2007

backing up your home folder

Here I outline the solution I chose for backing up my life, er... I mean home folder. (I'm sure there's life outside /home/tim somewhere...)

My requirements were:
  • backup to dvd+rw

  • >20GB of data to back up

  • no obscure formats (in case I don't have the backup tool when I need to restore)


I looked at several solutions for backups but ended up writing scripts to meet my needs.

The main script creates a tar file of my home directory, excluding certain items, which is then split into files suitable for writing to dvd+rw discs, with tar based verification, md5sums and file list text files created at the same time.

The reason for splitting to 3 files per disc is that the iso 9660 spec has a 2GB file size limit, and it's important that the discs are as simple as possible (ie no UDF) to aid recovery in awkward situations. This is also why I avoided compression.

backup_home.sh
#!/bin/bash -v
#DVD+R SL capacity 4,700,372,992 bytes DVD, (see wikipedia on DVD)
#ISO max file size 2GB. 4.38GB/3 = 1,566,790,997bytes = 1,494MB
#1,490MB to leave some space for listings and checksums
tar -cvv --directory /home tim --exclude-from backup_home_exclude.txt | split -b 1490m - /var/backups/tim/home/home.tar.split.
cd /var/backups/tim/home
md5sum home.tar.split.* > home.md5
cat home.tar.split.* | tar -t > home_file_list.txt
cat home.tar.split.* | tar -d --directory /home tim > home_diff.txt
ls -l home.* > home_backup_files.txt


backup_home_exclude.txt
tim/work*
tim/.Trash*
tim/.thumbnails*


This leaves me with a big pile of split files (named .aa, .ab etc) and a few text files. I proceeded to write 3 split files per disc, and put the 4 text files on every disc for convenience. I used gnome's built in DVD writing to create the discs.

I also wanted to verify the md5 checksums as the discs were created, so I wrote another little script to make life easier. This ensures the newley written disc has been remounted properly, and runs the md5 check. So long as the 3 relevant checksums came out correctly on each disc I can be reasonably confident of recovering the data should I need it.
"eject -t" closes the cdrom, which is handy.

reload_and_verify.sh
#!/bin/bash -v
cd /media
eject
eject -t
mount /media/cdrom
cd cdrom
md5sum -c home.md5
cd /media
eject


In addition to the above mechanism (which is a pain at best, mostly due to media limitations) I keep my machines in sync with unison which I strongly recommend for both technical and non-technical users. I gather it also runs on microsoft (who?), so you might find it useful if you are mid transition.

28 May 2007

My mum and her super ceramics

Thrown Handled Vase
Thrown Handled Vase,
originally uploaded by Sarah Abell.
My mum has started posting pics of her fab ceramics work on flickr. Go mum!

One of her early pieces has pride of place in my display cabinet and is admired by all. Watch this space for more funky designs as she heads towards the end of uni.

27 May 2007

Get emailed Tim's blog and photos

Can't be bothered to check here and see if I've written anything this month? Great news! You can now have my blog entries and latest flickr photos sent to you by email thanks to the feedblitz service.

You can subscribe to my blog using the email box on the bottom of the right hand menu, or by clicking here.

You can subscribe to my flickr photo feed here (public photos only - you have to be my flickr contact to see photos of friends I post).

starfighter


Whilst looking for backup packages on ubuntu via synaptic by searching for the word "tar", I stumbled across the package "starfighter". So I installed it. Then ran it. And played it. It's rather good fun. Ctrl to fire lasers, space to fire rockets, cursors to get around. Spend your time chasing enemy ships around the screen, and collecting bonuses / money. Has a decent sound track too.

Highly recommended. I love the open source world.



$ apt-cache show starfighter
Package: starfighter
Priority: optional
Section: universe/games
Installed-Size: 368
Maintainer: Debian Games Team
Architecture: i386
Version: 1.1-6
Depends: libc6 (>= 2.4-1), libgcc1 (>= 1:4.1.0), libsdl-image1.2 (>= 1.2.3), lib
sdl-mixer1.2 (>= 1.2.6), libsdl1.2debian (>> 1.2.7+1.2.8), libstdc++6 (>= 4.1.0)
, starfighter-data (= 1.1-6)
Filename: pool/universe/s/starfighter/starfighter_1.1-6_i386.deb
Size: 116320
MD5sum: 959f894e78517a3411c3c2656d61b85c
SHA1: ac7e2f458d4bd8c57056e11bb3da8609f35b528c
SHA256: 29c9adee1ee2fb52f1d790254683579e919655ad01bb806a02a59d32abcb8d58
Description: 2D scrolling shooter game
After decades of war one company, who had gained powerful supplying both
sides with weaponary, steps forwards and crushes both warring factions
in one swift movement. Using far superior weaponary and AI craft, the
company was completely unstoppable and now no one can stand in their
way. Thousands began to perish under the iron fist of the company. The
people cried out for a saviour, for someone to light this dark hour...
and someone did.
.
Features of the game:
.
o 26 missions over 4 star systems
o Primary and Secondary Weapons (including a laser cannon and a charge weapon)
o A weapon powerup system
o Wingmates
o Missions with Primary and Secondary Objectives
o A Variety of Missions (Protect, Destroy, etc)
o 13 different music tracks
o Boss battles
.
Homepage: http://www.parallelrealities.co.uk/starfighter.php
Bugs: mailto:ubuntu-users@lists.ubuntu.com
Origin: Ubuntu

10 April 2007

running partimage in batch mode

A continuation of the partimage project.

As it would appear that stdout support doesn't work due the user interface making use of stdout, I have been figuring out how to make the program run in batch mode, with a little help from KDevelop.

My continued findings:

The help presents a fully batch mode, -B
$ ./partimage --help
===============================================================================
Partition Image (http://www.partimage.org/) version 0.6.5_beta4 [stable]
---- distributed under the GPL 2 license (GNU General Public License) ----

Supported file systems:....Ext2/3, Reiser3, FAT16/32, HPFS, JFS, XFS,
UFS(beta), HFS(beta), NTFS(experimental)

usage: partimage [options] <action> <device> <image_file>
partimage <imginfo/restmbr> <image_file>

ex: partimage -z1 -o -d save /dev/hda12 /mnt/backup/redhat-6.2.partimg.gz
ex: partimage restore /dev/hda13 /mnt/backup/suse-6.4.partimg
ex: partimage restmbr /mnt/backup/debian-potato-2.2.partimg.bz2
ex: partimage -z1 -om save /dev/hda9 /mnt/backup/win95-osr2.partimg.gz
ex: partimage imginfo /mnt/backup/debian-potato-2.2.partimg.bz2
ex: partimage -a/dev/hda6#/mnt/partimg#vfat -V 700 save /dev/hda12 /mnt/partimg/redhat-6.2.partimg.gz

Arguments:
* <action>:
- save: save the partition datas in an image file
- restore: restore the partition from an image file
- restmbr: restore a MBR of the image file to an hard disk
- imginfo: show informations about the image file
* <device>: partition to save/restore (example: /dev/hda1)
* <image_file>: file where data will be read/written. Can be very big.
For restore, <image_file> can have the value 'stdin'. This allows
for providing image files through a pipe.

Options:
* -z, --compress (image file compression level):
-z0, --compress=0 don't compress: very fast but very big image file
-z1, --compress=1 compress using gzip: fast and small image file (default)
-z2, --compress=2 (compress using bzip2: very slow and very small image file):
* -c, --nocheck don't check the partition before saving
* -o, --overwrite overwrite the existing image file without confirmation
* -d, --nodesc don't ask any description for the image file
* -V, --volume (split image into multiple volumes files)
-VX, --volume=X create volumes with a size of X MB
* -w, --waitvol wait for a confirmation after each volume change
* -e, --erase erase empty blocks on restore with zero bytes
* -m, --allowmnt don't fail if the partition is mounted. Dangerous !
* -M, --nombr don't create a backup of the MBR (Mast Boot Record) in the image file
* -h, --help show help
* -v, --version show version
* -i, --compilinfo show compilation options used
* -f, --finish (action to do if finished successfully):
-f0, --finish=0 wait: don't make anything
-f1, --finish=1 halt (power off) the computer
-f2, --finish=2 reboot (restart the computer):
-f3, --finish=3 quit
* -b, --batch batch mode: the GUI won't wait for an user action
* -BX, --fully-batch=X batch mode without GUI, X is a challenge response string
* -y, --nosync don't synchronize the disks at the end of the operation (dangerous)
* -sX, --server=X give partimaged server's ip address
* -pX, --port=X give partimaged server's listening port
* -g, --debug=X set the debug level to X (default: 1):
* -n, --nossl disable SSL in network mode
* -S, --simulate simulation of restoration mode
* -aX, --automnt=X automatic mount with X options. Read the doc for more details
* -UX --username=X username to authenticate to server
* -PX --password=X password for authentication of user to server
===============================================================================


It is not immediately obvious what "X is a challenge response string" means.
I was able to get the program to run to a limited extend after a bit of searching the internet and trial and error with the option "-B x=y".

Having stepped through the program, it transpires that where I have put "x", the program expects a pattern to match with the title and content of any messages that would otherwise have been shown to the user, and "y" is the pre-programmed response. This is in the "interface_none" section.
"x" has to match the question in the form "message title/message content" and is compared using fnmatch which allows * as a wildcard (anyone got a good reference for fnmatch?).
If the program hits a question for the user, and cannot find a matching answer in the command arguments, "CInterfaceNone::invalid_programmed_response()" fires "exit(8)" and the program dies.

So far I have been running the program as a normal user, which will inevitably fail where it attempts to work with block devices / root owned files & folders. This produces a warning in the user interface, followed by program termination.

To bypass this first "not root" warning, I successfully used this pre-programmed answer:
./partimage -B Warning*=Continue
Alternatively the following is more specific and also works:
./partimage -B Warning*root*=continue

I haven't figured out how to pass more than one predefined answer in batch mode.

The run arguments can be set in KDevelop here:
project > options > debugger > program arguments

Side note:
The program has a base class of user interface defined, and then either instantiates interface_none or interface_newt depending on command line arguments.

If not using full batch mode it helps to set "enable separate terminal for application IO" in KDevelop (project > options > debugger) so that you can see the full user interface. However if the program exits then the console closes and any output is lost.

As part of stepping through the code, I came across a macro, which makes the program harder to follow while debugging due to not being able to step through. So I figured out what it did, and wrote out its output C++ code in full:

interface_none.cpp, line 103
#define MB_2(One,Other,ONE,OTHER)       \
int CInterfaceNone::msgBox##One##Other(char *title, char *text, ...) { \
char *result= lookup(title,text,"(unspecified)"); \
va_list al; \
va_start(al,text); \
message_only(#One "/" #Other, title, text, al, result); \
va_end(al); \
if (!strcasecmp(result,#One)) return MSGBOX_##ONE; \
if (!strcasecmp(result,#Other)) return MSGBOX_##OTHER; \
invalid_programmed_response(); \
return 0; \
}

MB_2(Continue,Cancel,CONTINUE,CANCEL)
MB_2(Yes,No,YES,NO)


my expanded version:
//notes: have expanded out macro so I can step through it.
int CInterfaceNone::msgBoxContinueCancel(char *title, char *text, ...) {
char *result= lookup(title,text,"(unspecified)");
va_list al;
va_start(al,text);
message_only("Continue" "/" "Cancel", title, text, al, result);
va_end(al);
if (!strcasecmp(result,"Continue")) return MSGBOX_CONTINUE;
if (!strcasecmp(result,"Cancel")) return MSGBOX_CANCEL;
invalid_programmed_response();
return 0;
}

int CInterfaceNone::msgBoxYesNo(char *title, char *text, ...) {
char *result= lookup(title,text,"(unspecified)");
va_list al;
va_start(al,text);
message_only("Yes" "/" "No", title, text, al, result);
va_end(al);
if (!strcasecmp(result,"Yes")) return MSGBOX_YES;
if (!strcasecmp(result,"No")) return MSGBOX_NO;
invalid_programmed_response();
return 0;
}


creating a ramdisk for testing.
http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
(I am on ubuntu 6.10 here, details may vary)

$ ls -l /dev/ram*
brw-rw---- 1 root disk 1, 0 2007-04-08 20:10 /dev/ram0
brw-rw---- 1 root disk 1, 1 2007-04-08 20:10 /dev/ram1
brw-rw---- 1 root disk 1, 10 2007-04-08 20:10 /dev/ram10
brw-rw---- 1 root disk 1, 11 2007-04-08 20:10 /dev/ram11
brw-rw---- 1 root disk 1, 12 2007-04-08 20:10 /dev/ram12
brw-rw---- 1 root disk 1, 13 2007-04-08 20:10 /dev/ram13
brw-rw---- 1 root disk 1, 14 2007-04-08 20:10 /dev/ram14
brw-rw---- 1 root disk 1, 15 2007-04-08 20:10 /dev/ram15
brw-rw---- 1 root disk 1, 2 2007-04-08 20:10 /dev/ram2
brw-rw---- 1 root disk 1, 3 2007-04-08 20:10 /dev/ram3
brw-rw---- 1 root disk 1, 4 2007-04-08 20:10 /dev/ram4
brw-rw---- 1 root disk 1, 5 2007-04-08 20:10 /dev/ram5
brw-rw---- 1 root disk 1, 6 2007-04-08 20:10 /dev/ram6
brw-rw---- 1 root disk 1, 7 2007-04-08 20:10 /dev/ram7
brw-rw---- 1 root disk 1, 8 2007-04-08 20:10 /dev/ram8
brw-rw---- 1 root disk 1, 9 2007-04-08 20:10 /dev/ram9


create and mount test ramdisk
# mke2fs /dev/ram0
# mkdir /media/ram0
# mount /dev/ram0 /media/ram0

add a test file and unmount the disk
# echo "test data #1." >> /media/ram0/foo.txt
# umount /media/ram0


the above, as a script:
#!/bin/bash
# create and mount test ramdisk
mke2fs /dev/ram0
if [ ! -d /media/ram0 ]; then
mkdir /media/ram0
fi
mount /dev/ram0 /media/ram0
#add a test file and unmount the disk
echo "test file." >> /media/ram0/foo.txt
date >> /media/ram0/foo.txt
cat /media/ram0/foo.txt
umount /media/ram0


Create & run script (as root, because it (un)mounts a file system, and creates a dir in a root owned folder):
$ gedit mkram.sh
$ chmod ug+x mkram.sh
$ sudo ./mkram.sh


Wierdly, partimage won't run in full batch mode without a second part to the -B switch, even if it's set up to not need to ask any questions. Supplying a dummy "x=y" seems sufficient to fool it.

Runing as root without asking for partition description works:
$ sudo ./partimage -d -B x=y save /dev/ram0 ram0.img


Restore image to a different ramdisk and check file:
$ sudo ./partimage -B x=y restore /dev/ram1 ram0.img.000
$ sudo mount /dev/ram1 /media/ram1
$ cat /media/ram1/foo.txt
test file.
Mon Apr 9 12:56:59 BST 2007

Success!

Script for checking file in saved partition:
#!/bin/bash
# mount and check restored ramdisk
if [ ! -d /media/ram1 ]; then
mkdir /media/ram1
fi
mount /dev/ram1 /media/ram1
cat /media/ram1/foo.txt
umount /media/ram1


To debug in KDevelop as root (in ubuntu):
alt-F2 (run)
gksudo kdevelop
open project... (go find existing copy)

So in summary, I have made progress in understanding the ways of this useful utility, and am a step closer to making a useful contribution to the project.

The rambling nature of this post reflects the way in which one begins to understand a new program. Hopefully it's not too hard to follow, or pick out the useful pieces. All feedback gratefully appreciated.

Tim.

30 March 2007

bad geek joke: the bourne shell

Here's one I made earlier (19 Oct 2006 according to my pc):
bourneshell.png
It's modified from http://thebourneidentity.com/, which is incidentally a film I very much like.

For those who don't know, this is the bourne shell:
http://en.wikipedia.org/wiki/Bourne_shell
Which is the father of BASH (/bin/bash, the Bourne Again SHell)

partimage + stdout, existing code

On first inspection it looks like some code already exists for writing an image to stdout (standard output).

image_disk.cpp, line 558
  if (strcmp(m_szImageFilename, "stdout"))
{
//... network output code hidden for clarity ...
}
else // it's stdout
{
m_fImageFile = stdout;
showDebug(1, "image will be on stdout\n");
}

Unlike stdin for restore, stdout for save is not currently available in the command line options. I did do a build earlier where I enabled it (which I don't have any more due to my build problems). I managed to pipe an image to hexdump and seemed to be able to see some of the user interface info in the output.

It would seem the problem with the stdout option is that even in batch mode the program outputs interface data to stdout, which then corrupts the image.

I think I shall attempt to remove all the UI stuff, and make it act more like all the other unix tools. Might also try to create a reusable library out of it.