Showing posts with label dev. Show all posts
Showing posts with label dev. Show all posts

10 September 2008

error "Not a valid object name" filtering an ex-svn git repo with renames

Under git version 1.5.4.3, which as of writing is the current in ubuntu 8.04 hardy heron, trying to split out a folder from a git repo where the repo was an import from subversion (svn), and the folder was renamed in the past causes a failure as show below:

#!/bin/bash -v
rm -rf test-case
mkdir test-case
cd test-case/
svnadmin create svnrepo
export repo="`pwd`/svnrepo"
mkdir -p import/a
echo '1' >> import/a/file.txt
svn import -m 'initial import' import file://$repo/trunk/
svn co file://$repo/trunk/ checkout
echo '2' >> checkout/a/file.txt
svn ci -m "file modified" checkout
svn mv -m "moving file" file://$repo/trunk/a file://$repo/trunk/b
svn up checkout
echo '3' >> checkout/b/file.txt
svn ci -m "modified again" checkout
svn log -v checkout
mkdir -p gitcopy/a
git svn clone file://$repo/trunk/ gitcopy/a
cd gitcopy/a
git filter-branch --subdirectory-filter a #fails [1]
cd ../../
mkdir gitcopy/b
git svn clone file://$repo/trunk/ gitcopy/b
cd gitcopy/b
git filter-branch --subdirectory-filter b


The marked line above fails with the following error:
[1] Rewrite bcfe73ef303832b6112a2419dc1da5f782672c14 (3/3)fatal: Not a valid object name bcfe73ef303832b6112a2419dc1da5f782672c14:a

This has been fixed in the latest build of git: version 1.6.0.1.294.gda06a and no longer fails.

The "fatal: Not a git repository" error message that filter branch produces doesn't seem to matter.

29 May 2008

sharing work between computers with a usb flash drive and git


I couldn't find anything exactly matching this on the net when I was figuring it out, so here's what I did.

This is working against a remote svn (subversion) server, but applies even without one.

On the first computer, grab your git working copy from svn with git-svn clone (or clone a git repo, or just start a new one).

mkdir ~/project.git
cd ~/project.git
git-svn clone svn://project-server/trunk
git repack #for good measure


Plugin your usb flash drive/stick/external harddrive, I'll presume it's a vfat/fat32/fat16 formated device mounted at /media/flash.
Create an empty repository on the drive, I'll use a bare one as there's no need to keep the working copy as well.

mkdir /media/flash/project.git
git --bare init /media/flash/project.git


Then add the flash drive git repo as a remote source in your local git repo. "flash" is the name I've given to the remote branch reference, you can call it whatever you like.

git remote add flash /media/flash/project.git

If you push immediately it will fail (as I discovered) because fat doesn't support the execute flag on files, so all the hooks are automatically active. I deleted all the hooks as I wasn't planning on using them, this may be wrong so no promises, but it seems ok so far for me. So remove the hooks with:

rm /media/flash/project.git/hooks/*

Then push your current local copy to the flash drive with:

git push flash

This will copy all your committed work onto the flash drive, even if you haven't pushed it upstream to the svn server with "git-svn dcommit" yet. Bonus! It won't copy any of your branches across though, so you if you want them you can add those independently with:

git push flash mybranch

Now move over to the second computer and plug the flash drive in. I'm making the same assumptions on paths and devices. Do another completely independent svn checkout as above:

mkdir ~/project.git
cd ~/project.git
git-svn clone svn://project-server/trunk
git repack #for good measure


Then add the flash drive's repo to the git repo on the second pc and pull all changes from the flash drive, optionally including any branches:

git remote add flash /media/flash/project.git
git pull flash master
git pull flash mybranch #if you like


When you've committed changes to git or pulled the latest changes from svn on either pc, you can then update the flash drive with the simple command:

git push flash

Which pushes all your changes on your master branch on to the flash drive. You are now ready to run the pull command on the other computer to get back in sync:

git pull flash master

If you don't push changes to the flash drive before committing to svn then things will be very simple. If you push changes to the flash drive, and then commit them to the svn server you will need to do a little more work. This is because when you run "git svn dcommit" it pushes your latest git commits to the svn server, deletes your locally committed changes, and then fetches them back from the svn server. This means that git won't recognise your local changes as being the same as the ones on the flash drive because they have different commit message and SHA1 hash. Attempting to push to the flash drive fails with the message "! [rejected] master -> master (non-fast forward)" as the old copy of the commits are still there.

To resolve this you need to throw away the matching set of changes on the flash drive. To do this you can use git reset as follows, where HEAD~1 should be the number of commits you need to throw away (eg HEAD~3 to throw away the last 3 commits that were pushed to the flash drive):

cd /media/flash/project.git
git --bare log #to see how many changes don't have svn information
git --bare reset HEAD~1


You can then push your changes as above.

cd ~/project.git
git push flash


I've glossed over subtleties with fetch vs pull, but hopefully you will find this useful.

This howto makes use of git's ability to pull from multiple sources, and I've found that git quite happily copes with changes that were checked in to svn coming via the flash drive, even when later running "git-svn rebase".

Please do comment or contact me with any problems, errors, extra info and feedback, and let me know if it was useful.

27 October 2007

OSS Contribution Number One!

I've had my first ever patch to an open source project accepted! Yay! (Fanfare please... no? anyone? oh well.)

Ok it's a one word change to a piece of documentation, but hey it's still something.

The project is gnucash. For details take a look at the bug report I created to hold the patch. I even got a thanks :-).

10 April 2007

running partimage in batch mode

A continuation of the partimage project.

As it would appear that stdout support doesn't work due the user interface making use of stdout, I have been figuring out how to make the program run in batch mode, with a little help from KDevelop.

My continued findings:

The help presents a fully batch mode, -B
$ ./partimage --help
===============================================================================
Partition Image (http://www.partimage.org/) version 0.6.5_beta4 [stable]
---- distributed under the GPL 2 license (GNU General Public License) ----

Supported file systems:....Ext2/3, Reiser3, FAT16/32, HPFS, JFS, XFS,
UFS(beta), HFS(beta), NTFS(experimental)

usage: partimage [options] <action> <device> <image_file>
partimage <imginfo/restmbr> <image_file>

ex: partimage -z1 -o -d save /dev/hda12 /mnt/backup/redhat-6.2.partimg.gz
ex: partimage restore /dev/hda13 /mnt/backup/suse-6.4.partimg
ex: partimage restmbr /mnt/backup/debian-potato-2.2.partimg.bz2
ex: partimage -z1 -om save /dev/hda9 /mnt/backup/win95-osr2.partimg.gz
ex: partimage imginfo /mnt/backup/debian-potato-2.2.partimg.bz2
ex: partimage -a/dev/hda6#/mnt/partimg#vfat -V 700 save /dev/hda12 /mnt/partimg/redhat-6.2.partimg.gz

Arguments:
* <action>:
- save: save the partition datas in an image file
- restore: restore the partition from an image file
- restmbr: restore a MBR of the image file to an hard disk
- imginfo: show informations about the image file
* <device>: partition to save/restore (example: /dev/hda1)
* <image_file>: file where data will be read/written. Can be very big.
For restore, <image_file> can have the value 'stdin'. This allows
for providing image files through a pipe.

Options:
* -z, --compress (image file compression level):
-z0, --compress=0 don't compress: very fast but very big image file
-z1, --compress=1 compress using gzip: fast and small image file (default)
-z2, --compress=2 (compress using bzip2: very slow and very small image file):
* -c, --nocheck don't check the partition before saving
* -o, --overwrite overwrite the existing image file without confirmation
* -d, --nodesc don't ask any description for the image file
* -V, --volume (split image into multiple volumes files)
-VX, --volume=X create volumes with a size of X MB
* -w, --waitvol wait for a confirmation after each volume change
* -e, --erase erase empty blocks on restore with zero bytes
* -m, --allowmnt don't fail if the partition is mounted. Dangerous !
* -M, --nombr don't create a backup of the MBR (Mast Boot Record) in the image file
* -h, --help show help
* -v, --version show version
* -i, --compilinfo show compilation options used
* -f, --finish (action to do if finished successfully):
-f0, --finish=0 wait: don't make anything
-f1, --finish=1 halt (power off) the computer
-f2, --finish=2 reboot (restart the computer):
-f3, --finish=3 quit
* -b, --batch batch mode: the GUI won't wait for an user action
* -BX, --fully-batch=X batch mode without GUI, X is a challenge response string
* -y, --nosync don't synchronize the disks at the end of the operation (dangerous)
* -sX, --server=X give partimaged server's ip address
* -pX, --port=X give partimaged server's listening port
* -g, --debug=X set the debug level to X (default: 1):
* -n, --nossl disable SSL in network mode
* -S, --simulate simulation of restoration mode
* -aX, --automnt=X automatic mount with X options. Read the doc for more details
* -UX --username=X username to authenticate to server
* -PX --password=X password for authentication of user to server
===============================================================================


It is not immediately obvious what "X is a challenge response string" means.
I was able to get the program to run to a limited extend after a bit of searching the internet and trial and error with the option "-B x=y".

Having stepped through the program, it transpires that where I have put "x", the program expects a pattern to match with the title and content of any messages that would otherwise have been shown to the user, and "y" is the pre-programmed response. This is in the "interface_none" section.
"x" has to match the question in the form "message title/message content" and is compared using fnmatch which allows * as a wildcard (anyone got a good reference for fnmatch?).
If the program hits a question for the user, and cannot find a matching answer in the command arguments, "CInterfaceNone::invalid_programmed_response()" fires "exit(8)" and the program dies.

So far I have been running the program as a normal user, which will inevitably fail where it attempts to work with block devices / root owned files & folders. This produces a warning in the user interface, followed by program termination.

To bypass this first "not root" warning, I successfully used this pre-programmed answer:
./partimage -B Warning*=Continue
Alternatively the following is more specific and also works:
./partimage -B Warning*root*=continue

I haven't figured out how to pass more than one predefined answer in batch mode.

The run arguments can be set in KDevelop here:
project > options > debugger > program arguments

Side note:
The program has a base class of user interface defined, and then either instantiates interface_none or interface_newt depending on command line arguments.

If not using full batch mode it helps to set "enable separate terminal for application IO" in KDevelop (project > options > debugger) so that you can see the full user interface. However if the program exits then the console closes and any output is lost.

As part of stepping through the code, I came across a macro, which makes the program harder to follow while debugging due to not being able to step through. So I figured out what it did, and wrote out its output C++ code in full:

interface_none.cpp, line 103
#define MB_2(One,Other,ONE,OTHER)       \
int CInterfaceNone::msgBox##One##Other(char *title, char *text, ...) { \
char *result= lookup(title,text,"(unspecified)"); \
va_list al; \
va_start(al,text); \
message_only(#One "/" #Other, title, text, al, result); \
va_end(al); \
if (!strcasecmp(result,#One)) return MSGBOX_##ONE; \
if (!strcasecmp(result,#Other)) return MSGBOX_##OTHER; \
invalid_programmed_response(); \
return 0; \
}

MB_2(Continue,Cancel,CONTINUE,CANCEL)
MB_2(Yes,No,YES,NO)


my expanded version:
//notes: have expanded out macro so I can step through it.
int CInterfaceNone::msgBoxContinueCancel(char *title, char *text, ...) {
char *result= lookup(title,text,"(unspecified)");
va_list al;
va_start(al,text);
message_only("Continue" "/" "Cancel", title, text, al, result);
va_end(al);
if (!strcasecmp(result,"Continue")) return MSGBOX_CONTINUE;
if (!strcasecmp(result,"Cancel")) return MSGBOX_CANCEL;
invalid_programmed_response();
return 0;
}

int CInterfaceNone::msgBoxYesNo(char *title, char *text, ...) {
char *result= lookup(title,text,"(unspecified)");
va_list al;
va_start(al,text);
message_only("Yes" "/" "No", title, text, al, result);
va_end(al);
if (!strcasecmp(result,"Yes")) return MSGBOX_YES;
if (!strcasecmp(result,"No")) return MSGBOX_NO;
invalid_programmed_response();
return 0;
}


creating a ramdisk for testing.
http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
(I am on ubuntu 6.10 here, details may vary)

$ ls -l /dev/ram*
brw-rw---- 1 root disk 1, 0 2007-04-08 20:10 /dev/ram0
brw-rw---- 1 root disk 1, 1 2007-04-08 20:10 /dev/ram1
brw-rw---- 1 root disk 1, 10 2007-04-08 20:10 /dev/ram10
brw-rw---- 1 root disk 1, 11 2007-04-08 20:10 /dev/ram11
brw-rw---- 1 root disk 1, 12 2007-04-08 20:10 /dev/ram12
brw-rw---- 1 root disk 1, 13 2007-04-08 20:10 /dev/ram13
brw-rw---- 1 root disk 1, 14 2007-04-08 20:10 /dev/ram14
brw-rw---- 1 root disk 1, 15 2007-04-08 20:10 /dev/ram15
brw-rw---- 1 root disk 1, 2 2007-04-08 20:10 /dev/ram2
brw-rw---- 1 root disk 1, 3 2007-04-08 20:10 /dev/ram3
brw-rw---- 1 root disk 1, 4 2007-04-08 20:10 /dev/ram4
brw-rw---- 1 root disk 1, 5 2007-04-08 20:10 /dev/ram5
brw-rw---- 1 root disk 1, 6 2007-04-08 20:10 /dev/ram6
brw-rw---- 1 root disk 1, 7 2007-04-08 20:10 /dev/ram7
brw-rw---- 1 root disk 1, 8 2007-04-08 20:10 /dev/ram8
brw-rw---- 1 root disk 1, 9 2007-04-08 20:10 /dev/ram9


create and mount test ramdisk
# mke2fs /dev/ram0
# mkdir /media/ram0
# mount /dev/ram0 /media/ram0

add a test file and unmount the disk
# echo "test data #1." >> /media/ram0/foo.txt
# umount /media/ram0


the above, as a script:
#!/bin/bash
# create and mount test ramdisk
mke2fs /dev/ram0
if [ ! -d /media/ram0 ]; then
mkdir /media/ram0
fi
mount /dev/ram0 /media/ram0
#add a test file and unmount the disk
echo "test file." >> /media/ram0/foo.txt
date >> /media/ram0/foo.txt
cat /media/ram0/foo.txt
umount /media/ram0


Create & run script (as root, because it (un)mounts a file system, and creates a dir in a root owned folder):
$ gedit mkram.sh
$ chmod ug+x mkram.sh
$ sudo ./mkram.sh


Wierdly, partimage won't run in full batch mode without a second part to the -B switch, even if it's set up to not need to ask any questions. Supplying a dummy "x=y" seems sufficient to fool it.

Runing as root without asking for partition description works:
$ sudo ./partimage -d -B x=y save /dev/ram0 ram0.img


Restore image to a different ramdisk and check file:
$ sudo ./partimage -B x=y restore /dev/ram1 ram0.img.000
$ sudo mount /dev/ram1 /media/ram1
$ cat /media/ram1/foo.txt
test file.
Mon Apr 9 12:56:59 BST 2007

Success!

Script for checking file in saved partition:
#!/bin/bash
# mount and check restored ramdisk
if [ ! -d /media/ram1 ]; then
mkdir /media/ram1
fi
mount /dev/ram1 /media/ram1
cat /media/ram1/foo.txt
umount /media/ram1


To debug in KDevelop as root (in ubuntu):
alt-F2 (run)
gksudo kdevelop
open project... (go find existing copy)

So in summary, I have made progress in understanding the ways of this useful utility, and am a step closer to making a useful contribution to the project.

The rambling nature of this post reflects the way in which one begins to understand a new program. Hopefully it's not too hard to follow, or pick out the useful pieces. All feedback gratefully appreciated.

Tim.

30 March 2007

partimage + stdout, existing code

On first inspection it looks like some code already exists for writing an image to stdout (standard output).

image_disk.cpp, line 558
  if (strcmp(m_szImageFilename, "stdout"))
{
//... network output code hidden for clarity ...
}
else // it's stdout
{
m_fImageFile = stdout;
showDebug(1, "image will be on stdout\n");
}

Unlike stdin for restore, stdout for save is not currently available in the command line options. I did do a build earlier where I enabled it (which I don't have any more due to my build problems). I managed to pipe an image to hexdump and seemed to be able to see some of the user interface info in the output.

It would seem the problem with the stdout option is that even in batch mode the program outputs interface data to stdout, which then corrupts the image.

I think I shall attempt to remove all the UI stuff, and make it act more like all the other unix tools. Might also try to create a reusable library out of it.

26 March 2007

compiling partimage

Had a problems getting partimage to compile on one of my pcs from a fresh checkout.
svn co https://partimage.svn.sourceforge.net/svnroot/partimage/trunk/partimage partimage

The ./autogen.sh script was failing as follows
tim@lap:~/projects/partimage$ ./autogen.sh
Running "autoreconf -vif" ...

autoreconf: Entering directory `.'
autoreconf: running: autopoint --force
autoreconf: running: aclocal -I m4 --output=aclocal.m4t
autoreconf: `aclocal.m4' is unchanged
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy --force
autoreconf: running: /usr/bin/autoconf --force
autoreconf: running: /usr/bin/autoheader --force
autoreconf: running: automake --add-missing --copy
configure.ac: 16: required file `./[config.h].in' not found
Makefile.am:1: AM_GNU_GETTEXT in `configure.ac' but `intl' not in SUBDIRS
automake: Makefile.am: AM_GNU_GETTEXT in `configure.ac' but `ALL_LINGUAS' not defined
autoreconf: automake failed with exit status: 1
Done.


Barked up lots of wrong trees, including looking for missing libraries, gettext config etc.

Turned out to be an old version of automake.

Not sure how my other pc ended up with the right version, but this pc's version was:
$ automake --version
automake (GNU automake) 1.4-p6


Installing new version (with some help from command line auto-completion):
$ apt-get install automake[tab]
automake automake1.5 automake1.8
automake1.4 automake1.6 automake1.9
automake1.4-doc automake1.7 automaken

$ sudo apt-get install automake1.9
...
$ automake --version
automake (GNU automake) 1.9.6


After updating automake, the ./autogen.sh script ran, and I could then run ./configure and make successfully, and was left with a binary for partimage in src/client/

Hurrah.

The solution came from a post by Tibor Simko on cdsware.cern.ch:

Re: problem with autoreconf when installing from cvs

* From: Tibor Simko
* Subject: Re: problem with autoreconf when installing from cvs
* Date: Thu, 18 Jan 2007 18:12:20 +0100

Hello:

On Thu, 18 Jan 2007, robert forkel wrote:

> $ autoreconf
> Makefile.am:23: AM_GNU_GETTEXT in `configure.ac' but `intl' not in SUBDIRS
> automake: Makefile.am: AM_GNU_GETTEXT in `configure.ac' but
> ALL_LINGUAS' not defined

Which version numbers of automake, autoconf, and gettext do you have?
E.g. automake versions prior to 1.9 and gettext versions prior to 0.14
will not work.

Best regards
--
Tibor Simko

13 March 2007

Today's project - partimage enhancement

me and partimage

I recently reorganised the partitions on my laptop, with the help of some invaluable OSS tools.

The laptop was split into OS partitions, with a data and swap partition at the end, but I'd starting running out of space. I have since made ubuntu my only OS at home, so no longer require multiple partitions.
My partition table ended up looking something like this: data | OS | more data | swap, and I wanted it to look like this: OS & data | swap, but without having to rebuild (again).

With another linux box available with bags of disc space, I did something like the following:
  • from each data partition and my home folder: tar -cv datafolder | ssh otherbox "cat > laptop/datafolder.tar", which gave me a tarball of all my data
  • boot into knoppix 4
  • use partimage to save os parition image into filesystem of another partition
  • scp osimage.img otherbox:laptop/
  • fdisk to set up new partitions
  • pipe the image back into partimage across the wire: ssh otherbox "cat laptop/osimage.img" | partimage .... plus some flags for batch writing to new partition
  • use parted (partition editor) to stretch partition image back up to full size of new partition.
  • fix grub with help from knoppix - hda(0,2) to hda(0,0) or something.
  • remove references to non existent partitions from fstab
Which was all great, but I feel there's a feature missing from partimage. Although it can read an image from stdin for writing to disc, it can't write an image to stdout from disc. This would have saved me some thinking and some hassle. So in the true spirit of OSS, I shall have a go at adding the functionality.

So far, I have grabbed the source from sourceforge's svn server, managed to compile the source (after being confused by a misleading error message) and installed an IDE. I started with Eclipse, as I've been using it a bit recently and really like it, but figure that perhaps the C++ devs aren't likely to be java fans and maybe they would choose something else. So I've installed KDevelop, and will be having a go with that.