Rob Golding

I just finished my last exam today - Web Programming and Scripting - which explains the distinct lack of activity around here in recent times. Thankfully I could end my exam season on a high, as web programming is, well, what I do - so it wasn’t too much of a challenge!

Something strange happens to me every time exams come around. I seem to pick up new projects, and just run with them. This time, I’ve become involved with a small group of people at university, writing a portal-style information system for universities. I suppose most people call this behavior procrastination, but I’m quite deeply in denial about that.

Ever since my post about the Backtrac Backup System, I’ve been really enjoying using Django. Something about it just makes developing for the web, well, exciting. That can only be good, right? I am the designated server administrator for this latest project, mostly due to the fact that I am the only one with a server to administer, and some of the things I’ve learned so far seem worthy of a mention here.  Firstly, we as developers were - how can I put it - stepping on each other toes somewhat. The project at this point had no version control, so we were just editing a bunch of files over SFTP. Obviously, some sort of Source Control Management was in order. I did some research, and decided that Git was a nice, modern alternative to the ever-popular SVN. It also meant that my server was constantly backed up by everyone on the team - but that’s just a bonus!

So, I installed Git, and started a repository. A lot of effort went into learning how the system works, and more importantly, how to make it work for us. Directed Acyclic Graphs thankfully made some sense to me, so I could just about understance the documentation. I wrote some custom hooks, and a C Program to syncronise the web-server. I was happy, and absolutely certain that this was the solution to all our woes. I was mistaken. Git just didn’t work the way I had hoped. The custom hooks were throwing permission errors all over the place, and my development team (read: my friend Rob Miles) was locked out of the repository. We made the decision yesterday to scrap Git, and go back to the previous system of editing the files over SFTP. We are always in constant communication when developing for the project, so it’s not too big a deal, but I feel that I failed as an adminisrator. You see, as an admin your job isn’t just to play with cool toys and loud servers - your primary purpose is to give the users what they need to work, and that is most certainly not what I achieved.

I’m glad I took the time to learn Git - and I’m sure it will help me later in life, in some way or another. It’s just unfortunate that it didn’t work out the way I had hoped for our project.

As a sort of follow-on from my last post, I thought I’d write a little about the latest little addition to my system - roaming Firefox profiles. This is something I had always considered to be almost impossible to achieve, without complicated logon and logoff scripts that syncronise the correct folder(s) to give the same effect as a ‘redirected’ profile. Well I discovered a much easier way to achieve actual *real* profile redirection, when browsing around the features offered by Group Policy Client Side Extensions.

The basic idea is that Firefox has a file called profiles.ini, which takes care of all the configured profiles, and where they are stored. I used this file to change the default profile location to within the user’s home directory on the file server. I had to use a home drive, mapped to the root of my users’ folder redirection directory on the server, as I presumed UNC paths were unsupported in the .ini file. I did this with a GP Preference drive map using the %USERNAME% variable, and an amazing feature of GP Preferences - parsing and even altering ini files. You can specify which section of the ini file you are interested in, and which key you want changing. How useful! My policy looks like this:

You can probably work out what’s going on here, but I’ll give a quick overview. Basically, you specify the ini file to edit - in this case it’s %APPDATA%\Mozilla\Firefox\profiles.ini. Using the %APPDATA% variable means that it will always resolve to the correct location in the user’s local profile, whether they are on XP or Vista. Then we specify the section of the ini file - I’m interested in Profile0. This is the only profile present by default, but allows users to have multiple profiles if they wish without overwiting their settings when they log off. Finally, you specify the key to change, and what to change it to. I rename the default profile to Firefox, and change it’s location to H:\Firefox. Also, this path is not relative, so I have to change IsRelative to 0.

OK, so now we have the Firefox profile location sorted, we need to make sure that folder exists - or Firefox will just overwrite our changes and make it’s own folder in the default location. It’s easy to use GP Preferences for this as well - as there’s a Folders preference category. So I just made a new folders preference for \\zeus\UserData\%USERNAME%\Firefox, with the action of create (zeus is my main DC and File Server). I used the UNC path to be sure that the folder is created, even if the drive map hadn’t come into effect when this preference was applied. Also, a little trick I had to pull here was ticking the checkbox titled Run in logged-on user’s security context on the common tab. This is because only the user has permissions on their home directory, so this preference needed to run in the context of that user for it to work successfully (without access denied errors).

Once this was finished, the system started to work flawlessly. I copied the contents of existing Firefox profiles to the newly created directories, and they were picked up by Firefox with no problems. New users get blank profiles as expected, but they are stored on the file server instead of the local machine. One little issue I have encountered is that a user can’t logon at more than one machine, and start Firefox - as the program can’t lock particular files in the profile. This just results in a message saying this Firefox is already running though, which is pretty much correct (and I can’t see why this would ever cause problems for the user).

The last trick I employed, to make things a little speedier and to reduce uneccesary file server traffic, was to disable disk caching on the roaming Firefox profiles. To do this, I used a file policy in GP Preferences to copy a tiny user.js (Firefox’s preference file override) which contained only one line:

user_pref(”browser.cache.disk.enable”, false);

This turns off disk caching completely, which will not only save space on the file server, but should speed things up as well. I hosted this file elsewhere on the file server, and told the file policy to simply copy it into place, within the user’s Firefox profile. Here’s the preference:

So there you have it, my technique for enabling roaming Firefox profiles. If you’ve achieved the same through a different method, or have any ideas on this this could be improved, then I’d love to hear how - feel free to comment on this post.

Thanks to the MSDNAA program, I’m able to try out the latest version of Windows Server in the lab. I opted to migrate my domain accross to a new machine, instead of performing an in-place upgrade. Personally I feel this is a much safer bet, and tend to migrate domain controllers whenever I’m doing something pretty major to a DC.

So far everything looks good, I’ve upped the forest and domain functional level to Server 2008, so I can now take advantage of some of the new features - though I’m yet to find out what they all are! The best thing so far (by a mile I’d like to add) is the addition of Group Policy Preferences. Although it’s annoying having to install the Client-Side Extensions on every machine in the domain (that is if WSUS isn’t in use), the gains faw outweigh this bit of pain. I only wish an MSI could have been released, so that it could easily be pushed out using the existing Group Policy infrastructure. Never mind, eh?

Anyway, on with the good! The new GP Preferences allow an administrator to define, amongst others, drive maps for client machines, printer connections and power options. As you may be thinking, this just about does away with the need for logon scripts! Most, if not all of the common tasks that are performed with logon scripts can now be done from a group policy object.

There are also a lot of changes to the way Active Directory works. In Server 2008, Active Directory Domain Services can be installed on a machine, without actually making it a DC. What this means is that a standard server build can be ’sysprepped’ with the files required for promoting the server to a DC, without actually doing the promotion. Also, Read-Only Domain Controllers (RODCs) have been introduced as a new feature. Essentially, an RODC just caches queries from a normal DC, usually located at another site - apparently allowing for faster logon times at remote sites with slow links. After discussion with a colleague, however, the benefits of such a system are maybe not quite as advertised. For example, only one RODC can be installed per site - so larger sites can’t benefit from the redundancy and load balancing offered by multiple DCs, if RODCs are used. Also, the much-touted security advantages of using an RODC aren’t as they seem either, as the database can be just as easily written to, just through another “normal” DC.

More on this later!

My life has been pretty busy as of late, mostly with a new project I am working on called BackTrac. I am developing a network backup solution, written entirely in Python - using the Django framework as a front-end web interface.This came at quite a good time, as I’ve been asked to give a presentation on any highly technical topic, on which I know my stuff. The plan is to use BackTrac as a base, and to explain the technical concepts behind the system that make it work. The things I am going on concentrate on are:

• Pyhon in general
• XML-RPC
• Filesystem hardlinks
• The MVC concept, and Django

This should be enough content to fill a 15-minute slot, I hope.

Now I’ll explain a little bit about how BackTrac works, for the benefit of those not coming to the presentation.

What I wanted to achieve with BackTrac was a smart backup system, that doesn’t necessarily have to be the fastest. I wanted a very detailed web interface, with scheduling capabilities and log view. I decided on Django for this, seeing as I had already decided on Python for the system itself.

The nodes in the system use XML-RPC to communicate, and SMB for the actual file-copying. The real bonus of this system is that it takes advantage of a little-understood feature, present in most modern filesystems, called hardlinks. Hardlinks enable the user to essentially point to the same piece of data on a disk from more than one position. Essentially the concept is that if a file has been backed up before, why back it up again? Just create a link or “shortcut” (a hardlink) to the previously backed-up file. This is the basis for BackTrac.

Also, Django is turning out to be a real treat. This is the first time I have used an MVC like Django, and I’m really enjoying it. Web development is exciting again! I do agree with one point however, that Django makes the easy things easy, and the hard things impossible. Not quite impossible in my case, but I’ve had to do some pretty strange things to get Django to do what I want.

Finally, I’ve had the BackTrac project approved on Sourceforge.net, so watch out - I’ll be doing the initial import soon and getting some web content online. First though, I have to decide on the best way to distribute the application. This is a bit more complicated than it might otherwise be, because there are three different aspects of the system - the client application, the server application, and the Django project. On this topic or any other, as usual, opinions are most welcome.

To go with the recent network upgrades and anti-spam system, I have been working on a new way to back all this information up. The solution I’ve come up with is surprisingly simple: VSS Snapshots with Robocopy to mirror the changes. The basic idea is that the backup script creates a Volume Shadow Copy Service Snapshot, and “exposes” (mounts) the snapshot with an unused drive letter. Robocopy then mirrors the contents of this snapshot to the backup drive, allowing even files that are locked to be backed up. Add in a bit of error-checking and status emails, and we have a pretty solid backup system. I’ll run through the details below.

To create the VSS snapshot, I used a script sourced from an MSDN blog called CreateShadow, which I modified slightly to suit my purpose. I had it keep the temporary variables script, so I could use it later on (once the backup has finished) to delete the snapshot.

Once the snapshot is created and exposed, I used Robocopy with the mirror (/MIR) switch, to copy the contents to the backup drive. It just so happens that the backup drive is connected to a Samba server running on Ubuntu. This meant that I ran into a problem with timestamps whereby files were always classified as “newer”, even if they hadn’t changed at all since the last run. I fixed this by using the Fat File Times (/FFT) switch which gives a 2-second granularity on the timestamp of files, which solved the issue straight away.

The backup having completed, the script calls the temporary variables script generated by the CreateShadow script, to reinstate the snapshot ID, which is then used to remove the shadow copy cleanly.

In theory, this is an extremely efficient and robust backup system - not to mention being completely free of any licence fees. I may improve it in the future by adding functionality with multiple backup sets - at the moment I only have one day to recover from any accidental deletions - barring the previous versions.

One thing I am stuggling with at present, however, is the fact that when the backup runs under scheduled task at 3am, a number of files throw access denied errors - namely any files or directories with special characters. This is a particularly strange issue as the process works flawlessly when launched manually. I am still trying to solve the issue, but I’ll be sure to post an update if and when I find the solution.

Recently I have been having a strange issue on my desktop PC, whereby saving an Office document (Word, Excel etc.) inside the My Documents folder - which is redirected to my file server - gives the following error:

There has been a network or file permission error. The network connection may be lost.
(<filename>)

I’ve been trying to find out the cause of this for some time now, while working round it by simply saving the file to my desktop and copying it over to My Documents.

Anyway, I have just found the problem. I had a little plugin installed that allowed the indexing service on Windows Vista to index a network location, meaning I could search the My Documents folder quickly. This was the cause of the issue, and removing the index solved the problem. I wanted to write the solution up here in the hope that this helps someone in the same situation.

Note: There have been other solutions to this problem cited, including anti-virus programs, and network congestion.The KB article for this issue is located here.

My home network has been growing and growing ever since the start. At first, just little linux firewall, and it’s not done yet.

I’ve recently upgraded the my new server, adding another 4GB of RAM to bring the total to 8GB - to give me some more room for playing around. Also, I’ve decided that with all that extra memory I can upgrade to VMware Server 2.0 safely, and all seems to have gone well. Also surprisingly, the load on the machine hasn’t gone up, even with the addition of 2 new virtual machines.

The first new server is an anti-spam gateway for my Microsoft Exchange organisation, and is working flawlessly so far. Second is a Windows Server 2008 machine, which I plan to migrate the domain onto some time in the future. I plan to do some work with the brand new O/S, and see what’s what. I’m sure there will be plenty of material to keep up-to-date with, so keep checking back!

My new laptop, a Dell Studio 1535, has started exhibiting a strange problem as of late; when I hibernate it over night, it resumes early in the morning, waking me up with the fan spinning.

I was preparing to send the thing back to Dell, when it happened one night and I checked out the event logs. It turns out that Windows Update was resuming the machine from hibernate - and then updating itself. After turning this off, I’ve had a good night’s sleep! Also, Windows Defender was scheduled to scan the machine every morning at 2.00am, so that’s been disabled too.

-

I hope this post can provide answers to anyone else experiencing a similar issue.

In my last post, I talked about configuring Ubuntu for use with different proxy servers, and something became quite evident. It would have been nice to have a way of aliasing the IPs of those servers, so instead of typing the number out, one could simply type “proxy”, for example.

This can be easily achieved using the hosts file, which is present on both linux and windows operating systems. Here I will discuss how to use the hosts file in Ubuntu linux.

First, you must gain access to the file. It can only be written to as root, so the sudo command must be used, in conjunction with your favourite editor. For example:

sudo gedit /etc/hosts

As you can see, the hosts file is located at /etc/hosts.

To add entries to the hosts file, simply add another line at the end, following the president set by the one or two lines alreay present. The syntax is basic, and consists of the IP to be aliased, then the name that you would like to alias it with. For example, the proxy in my last post could be entered as follows:

128.243.253.119    proxy

As it happens, this is actually a surprisingly effective method of blocking malicious websites, and even adverts. Quite simply, any domain that you would not like your browser to access, can be added to the hosts file with an IP of 127.0.0.1. This is the IP address for the local machine you are currently working on. For a huge list of such websites, check out www.someonewhocares.org/hosts. Simply copy and paste the file on that site into your hosts file, and you’re set. It should be noted that this will work on windows, Mac or linux. On Windows XP/Vista the hosts file is located in:

C:\Windows\system32\drivers\etc\hosts

Obviously if your Windows installation is on a different drive, simply replace C: with the relevant letter.

I have recently enrolled on a Computer Science course at The University of Nottingham, and as such have had to make sure my machines correctly use their proxy server for web access. This post outlines the process of configuring Ubuntu for exactly that purpose - and could be applied to any network with a similar layout.

Network Proxy

First and foremost, Ubuntu has a setting in gnome for the Network Proxy, which should set gnome’s proxy - but I can’t see as it affects anything at all - still, better to be safe than sorry.

At Nottingham University, the recommended configuration is a proxy auto-configuration script (proxy.pac) which is downloaded by the client and parsed to configure the appropriate proxy server. In this case, it is located at http://wwwcache.nottingham.ac.uk/proxy.pac. This URL is entered into the correct field of the gnome Network Proxy settings dialogue.

Terminal Proxy

Secondly, the terminal has a proxy configuration option, so that programs that run inside the terminal making HTTP requests can access the internet - namely wget and aptitude. This is slightly more difficult to configure than the previous, and is achieved like so:

The terminal proxy is set using a variable called http_proxy, which is set using the export command, i.e.

export “http_proxy=http://proxy_server_ip:port”

In this case, the proxy server’s IP and port for the SNS (Student Network Service) is 128.243.253.119:8080.

This change can be made permanent by editing /etc/bash.bashrc, and adding the above line to the end of the file. Otherwise, the change is only effective in the terminal window currently open by the user, and disappears when it is closed.

Synaptic

Lastly, Synaptic Package Manager must have the proxy set, in order to update your installation using the in-built Update Manager or Synaptic GUI. This is done by opening Synaptic, and choosing Settings, Preferences, and setting the above proxy information using the Network tab. Unfortunately Synaptic cannot read auto-configuration scripts, so the IP and port must be manually entered here.

Obviously once all this is done, Firefox must be configured to use the correct proxy - but I trust you know how to do that! This turns out to be quite a pain, so I’ll be looking at ways to do this in one fell swoop. If anyone has any suggestions, then please let me know.

Note: These changes can be made much less painful by adding an entry to the hosts file for each of the proxy IPs you need to configure - so you only need to type the word proxy for example, instead of the entire IP. I will document this process shortly in a separate post.