backing up Vista

So here's a tale of annoying things.

I generally try and avoid running anything proprietary at home, and especially anything from Microsoft. But for reasons beyond my control a copy of Vista has embedded itself in our household.

For my linux backups I've settled on dar (Disk ARchiver) + dargui as it applies the keep it simple rule (certainly in comparison with a lot of other tools I tried, such as the ubuntu default "home user backup" tool, which has a simple ui but offers limited control and I'm not sure how easily I could recover files from it in a disaster). Dar is like tar (Tape ARchiver) but more designed with backup to disk in mind, which is what I was after in order to backup to usb hdd, and most importantly does incremental/differential backups the way I want.

So then I came to back up this Vista home directory (sorry, Users directory / profiles). I have disliked the old fashioned microsoft backup files (.bkf) from Windows XP and before ever since I tried to get files out of one from a Linux box. Turns out it's not exactly the best supported format. I didn't have much luck with mtftar. It seems Microsoft have produced a replacement (plus ça change) to the old windows backup that is evidently designed to be simple, which has very strange ideas about how you might want to back up your pc. It seems keener to back up the sample image files from the office install than than the user's photos. I very quickly fell out with this tool and moved on.

Some people might recommend windows home server, but I am not about to pay for more shoddy Microsoft software in order to solve problems created by other shoddy Microsoft software. Vote with your wallet, as they say.

So next on the list, remote backup from a linux box. backup pc looked great, and has many nifty features, however having got it all set up I got permissions errors in the My Documents etc folders, which are the important ones. I tried different user permissions for the backup user, and different group memberships, though stopped short of resetting all user directory permissions so as to not break anything but couldn't get past these errors. More details on that attempt in my backuppc and windows vista blog entry. There is a hint in the Robocopy wikipedia that there is some special mode needed to be able to get past these permissions issues.

"The so-called Backup mode is an administrative privilege that allows Robocopy to override permissions settings (specifically, NTFS ACLs) for the purpose of making backups."

But I didn't get any further than that.

So finally I come to the conclusion that Vista just doesn't want you to do backups without paying microsoft more money, and that they have forgotten or never knew the KISS mantra that makes *nix such a pleasure to work with. (Rather opting for their usual "making simple things easy and difficult things impossible".)

It was Linux Format 127 Ubuntu 9.10 Cover Disc that came to the rescue. Popped the disc in,rebooted, connected my usb hdd (formatted with ext3 of course), ran apt-get install dar, opened the disk icon on the desktop representing the evil vista installation partition on the local disk (to get it mounted), opened the usb hdd disk icon (also to get it mounted), then ran dar -c /media/usbhdd/backups/vistargh -z from the directory /media/vista/Users/. This ran fine and I was able to read the file from a better operating system with no issues.

MCTS in ASP.NET 3.5 - Mission accomplished.

I am now a Microsoft Certified Technology Specialist in .NET Framework 3.5, ASP.NET Applications, which is nice.

getting supybot to announce new bugzilla bugs

getting supybot to announce new bugzilla bugs - I've just put here the key non-obvious things that tripped me up when trying to set this up.

All done on Ubuntu 8.04.3 LTS

Install supybot and the supybot bugzilla plugin.

Create a system group (supybot) and user (bugbot) to run supybot as.

Set up your supybot configuration file as desired.

Getting supybot to start at startup:
http://www.schwer.us/journal/2005/04/17/supybot-init-script-for-debian/

Here's my modified init script

$ cat /etc/init.d/bugbot
#! /bin/sh
#
# supybot init script
# http://www.schwer.us/journal/2005/04/17/supybot-init-script-for-debian/
#

PATH=/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/supybot
NAME=supybot
DESC=supybot

test -f $DAEMON || exit 0

set -e

case "$1" in
start)
echo -n "Starting $DESC: "
start-stop-daemon --start --quiet \
--chuid bugbot --exec $DAEMON -- --daemon /etc/supybot/bugbot.conf
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
start-stop-daemon --stop --quiet \
--oknodo --exec /usr/bin/python
echo "$NAME."
;;
restart)
$0 stop
$0 start
;;
*)
echo "Usage: $0 {start|stop|restart}" >&2
exit 1
;;
esac

exit 0

Set up bugzilla to send a copy of all bugmail to a local address (eg bugbot@localhost), and configure exim4 to accept local mail (as well as smart host delivery), using the mbox format.

sudo dpkg-reconfigure exim4-config

Start a conversation with bugbot, get it to identify you, then set the required configuration by sending it messages (you can also set these in the supybot .conf file for your bot):

• config plugins.Bugzilla.mbox /var/mail/bugbot


• config plugins.Bugzilla.bugzillas.your-bugzilla-name.watchedItems.all True
  which will turn on the announcements (i had to read the code to find that one!)

Note that supybot doesn't immediately write config changes to disc.

bugzilla upgrades and user tokens

It's bugzilla upgrade time for my private install, and I have for the second time run into a strange issue with the tokens system. Since this is the second time and I know how to fix it, here it is for the record.

I have upgrade from v3.0.4 to 3.4.3.

Once the site was up again, saving the site parameters (editparams.cgi) showed a big red warning:

It looks like you didn't come from the right page (you have no valid token for the edit_parameters action while processing the 'editparams.cgi' script). The reason could be one of:

• You clicked the "Back" button of your web browser after having successfully submitted changes, which is generally not a good idea (but harmless).
• You entered the URL in the address bar of your web browser directly, which should be safe.
• You clicked on a URL which redirected you here without your consent, in which case this action is much more critical.

Are you sure you want to commit these changes anyway? This may result in unexpected and undesired results.

[Confirm Changes]

Or throw away these changes and go back to editparams.cgi.

Pushing the button doesn't work (same page shows again).

After much digging last time I discovered that the tokens it refers to are stored in table bugs.tokens, and that the size of the field is wrong in my installation after the upgrade (again).

mysql> use bugs;
mysql> describe tokens;
+-----------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| userid | mediumint(9) | YES | MUL | NULL | |
| issuedate | datetime | NO | | NULL | |
| token | varchar(5) | NO | PRI | NULL | |
| tokentype | varchar(8) | YES | | NULL | |
| eventdata | tinytext | YES | | NULL | |
+-----------+--------------+------+-----+---------+-------+
5 rows in set (0.02 sec)

According to the published schema, token should be varchar(16). http://www.ravenbrook.com/project/p4dti/tool/cgi/bugzilla-schema/index.cgi?action=single&version=3.4.2&view=View+schema#table-tokens

To fix the problem I modified the data type as follows:


mysql> alter table tokens modify column token varchar(16) not null;
Query OK, 20 rows affected (0.32 sec)
Records: 20 Duplicates: 0 Warnings: 0


And then I was able to change my parameters.

https://bugzilla.mozilla.org/show_bug.cgi?id=527780

life update - software learning

I'm currently reading Journey of the Software Professional: Sociology of Computer Programming. It's proved so far to be a unique insight into how developers go from seeing a problem to providing a solution, good or bad. It's quite hard going as there's a lot of detailed analysis into how we work, but it's certainly given me food for thought.

In another area of self improvement, I recently passed 70-536: TS: Microsoft .NET Framework - Application Development Foundation, and am currently reading up on the ASP.NET exam with a view to gaining the MCTS cert. Much as I had hoped I'd left exams behind years ago, it is proving to be a useful exercise in filling in the gaps in my knowledge. Today I finally got around to creating something with Web Parts, which I wouldn't otherwise have done, another plan filed away in my cognitive library (cf. Journey of the Software Professional).

backuppc and windows vista

Steps I took to get a backuppc server on an ubuntu 8.10 pc in order to be able to back up a windows vista business pc. I've only documented the non-obvious and undocumented items here.

• Added a user called backup to the vista pc.
• Added the backup user to the "administrators" group. I tried the "backup operators" group but it didn't give access to the user profiles. It looks like robocopy has some black magic that allows it to bypass ACLs when a member of backup operators.
  
• Enabled admin shares (ie \\machine\C$) by adding DWORD registry key "LocalAccountTokenFilterPolicy" to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System and setting the value to 1. See How to access Administrative Shares on Vista (C$) by PaulSpoerry
  
• Removed the "-N" from the SmbClientFullCmd etc options in backuppc's transfer (xfer) settings as it didn't seem to be using the provided password. Troubleshooting was aided by running "/usr/share/backuppc/bin/BackupPC_dump -v -f vistamachine" directly as user backuppc.
• Hide the backup user from the welcome screen by adding DWORD HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\SpecialAccounts\UserList\backup with value 0 to the registry. "backup" is the name of the user to hide, 0 means hide, 1 means show.
  

enjoy.

See also:

• ubuntu backuppc/vista guide

Current status: This has only been partially successful so far. The backup runs but has lots of files missing due access denied errors on many of the important folders. It looks like I will have to manually give "backup operators" permissions to these folders. Sigh. Vista is proving to be less than easy for me to support.

git-svn imports for open source projects

I've just written a lengthy open letter to github in the hope that they will support continuous centralized imports from svn to git.

If they don't wish to rise to the challenge (and I wouldn't hold it against them), I am wondering if it would be viable to run a server on fasthosts or something funded by donations that would do nothing but continuously import from svn servers, and push the result to github.

That's all for now.

mini police eye in the sky

Just a quick note on an idea that's been bothering me.

How about a mini semi autonomous police surveillance helicopter?

Would cheaper than a real helicopter, but still offers many of the advantages of having an eye in the sky that can go direct to the scene without contending with traffic etc.

Something like a electric model helicopter with 2+ blades for stability, with a petrol generator for longer run times.

Electronics and gyros to make it naturally stable, and a gps & altimeter for auto navigation.

Carrying a payload of normal and infrared cameras for information gathering.

Transmitting a live video feed over long range radio, and offering long distance control, so it could be controlled centrally by an operator.

I reckon these could be produced for a few thousand pounds each and provide excellent assistance to the force.

format all documents in a visual studio solution

Here's a handy macro script for visual studio I knocked together today.
It runs "edit, format document" on every document of the listed file types.

You have to keep an eye on it as it's interactive and does sometimes pop up a message and wait for an answer.

You can get the vb file at http://github.com/timabell/vs-formatter-macro
More info at http://wiki.github.com/timabell/vs-formatter-macro

Below is the original code. Note that this is older than the version available on github above.

Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Collections.Generic
Imports System.Diagnostics
Imports System.Text

Public Module Formatting

 Dim allowed As List(Of String) = New List(Of String)
 Dim processed As Integer = 0
 Dim ignored As Integer = 0
 Dim errors As StringBuilder = New StringBuilder()
 Dim skippedExtensions As List(Of String) = New List(Of String)

 Public Sub FormatProject()
  allowed.Add(".master")
  allowed.Add(".aspx")
  allowed.Add(".ascx")
  allowed.Add(".asmx")
  allowed.Add(".cs")
  allowed.Add(".vb")
  allowed.Add(".config")
  allowed.Add(".css")
  allowed.Add(".htm")
  allowed.Add(".html")
  allowed.Add(".js")
  Try
   recurseSolution(AddressOf processItem)
  Catch ex As Exception
   Debug.Print("error in main loop: " + ex.ToString())
  End Try
  Debug.Print("processed items: " + processed.ToString())
  Debug.Print("ignored items: " + ignored.ToString())
  Debug.Print("ignored extensions: " + String.Join(" ", skippedExtensions.ToArray()))
  Debug.Print(errors.ToString())
 End Sub

 Private Sub processItem(ByVal Item As ProjectItem)
  If Not Item.Name.Contains(".") Then
   'Debug.Print("no file extension. ignoring.")
   ignored += 1
   Return
  End If
  Dim ext As String
  ext = Item.Name.Substring(Item.Name.LastIndexOf(".")) 'get file extension
  If allowed.Contains(ext) Then
   formatItem(Item)
   processed += 1
  Else
   'Debug.Print("ignoring file with extension: " + ext)
   If Not skippedExtensions.Contains(ext) Then
    skippedExtensions.Add(ext)
   End If
   ignored += 1
  End If
 End Sub

 Private Sub formatItem(ByVal Item As ProjectItem)
  Debug.Print("processing file " + Item.Name)
  Try
   Dim window As EnvDTE.Window
   window = Item.Open()
   window.Activate()
   DTE.ExecuteCommand("Edit.FormatDocument", "")
   window.Document.Save()
   window.Close()
  Catch ex As Exception
   Debug.Print("error processing file." + ex.ToString())
   errors.Append("error processing file " + Item.Name + "  " + ex.ToString())
  End Try
 End Sub

 Private Delegate Sub task(ByVal Item As ProjectItem)

 Private Sub recurseSolution(ByVal taskRoutine As task)
  For Each Proj As Project In DTE.Solution.Projects
   Debug.Print("project " + Proj.Name)
   For Each Item As ProjectItem In Proj.ProjectItems
    recurseItems(Item, 0, taskRoutine)
   Next
  Next
 End Sub

 Private Sub recurseItems(ByVal Item As ProjectItem, ByVal depth As Integer, ByVal taskRoutine As task)
  Dim indent As String = New String("-", depth)
  Debug.Print(indent + " " + Item.Name)
  If Not Item.ProjectItems Is Nothing Then
   For Each Child As ProjectItem In Item.ProjectItems
    taskRoutine(Child)
    recurseItems(Child, depth + 1, taskRoutine)
   Next
  End If
 End Sub

End Module