Today, I finally solved the issue. It turns out that something to do with the m4 package was causing the error, with the following output:

MozPlugger: Error: Failed to execute m4.
Mozplugger: M4 parsing of config generated error

So, I checked out the package in question, and found that it wasn't installed. To my great relief (and some frustration), a simple

sudo apt-get install m4

fixed the problem instantly.

I only hope that this post solves this extremely annoying issue for someone else.

• The Django server (yet to be named)
• The client daemon (backtracd)
• The backup library itself (backuplib)

As I mentioned in the previous post, the Django server will be based around Celery. This will offer me the ability to schedule a task for a certain date and time in the future, which is a huge part of the implementation taken care of.

One thing I am yet to figure out, however, is how the server will communicate with the client daemons. At the moment I'm thinking XML-RPC, but it seems to me that it will end up a sort of mish-mash of different technologies - as the Django server will be exposing a JSON API (probably via Piston). I have also considered running Django on the clients too, so everything can use JSON/Piston, but this seems at the moment like an unnecessary burden.

Another thing to consider is the data that is to be transmitted between the nodes. Should I use a bunch of fields in a JSON object to represent the success/failure of a job, and how much data was backed up, etc.? Another option is to pickle a Result object of some sort - representing a result as a Python instance to be interpreted by the server. These are the things I love to think about when designing a system, and I'll certainly have plenty to write about in the inevitable report.

Something I can work on in the meantime, however, is backuplib - the client library that will actually copy the data. At the moment, it's just a wrapper around rsync/tar. That probably won't change, but it will need to get a bit smarter in the way it handles failures (and restores, of course) if it's going to make for the basis a decent backup system.

Beginnings

Firstly, I use virtualenv to keep the Python environments for the different projects separate, but that's a topic in itself. This isn't a requirement, but it helps if you have dependencies that you don't want installed system-wide, or projects that use different versions of a particular package.

You can read more about virtualenv and Django here.

The directory structure for my webapps directory (which lives inside /home/) looks something like this:

webapps/
|-- apache
|   |-- coral.conf
|   |-- coral.wsgi
|   |-- myuni.conf
|   `-- myuni.wsgi
|-- coral
|   |-- env
|   `-- repository
|-- myuni
|   |-- env
|   `-- repository

The two projects shown here are Coral Issue Tracker and MyUni. The directory for each project contains the virtual env for that project, and the repository. Alongside the project folders is a directory called apache, which holds two files for each project: the WSGI file (<project-name>.wsgi) and the Apache configuration file (<project-name>.conf).

Apache Configuration

Then, in the main Apache configuration, I use a directive that includes all these files automatically:

Include    /home/webapps/apache/*.conf

The individual project configuration files then contain a set of simple directives almost straight from the Django deployment documentation. For example, the MyUni configuration looks like this:

WSGIScriptAlias  /myuni        /home/webapps/apache/myuni.wsgi
Alias            /myuni/docs   /home/webapps/myuni/repository/docs/_build/html
Alias            /myuni/media  /home/webapps/myuni/repository/myuni/static

WSGI Files

The WSGI files are also pretty simple, though there are a couple of things worth explaining. Here is the WSGI file for the MyUni project:

import os
import sys
import site
 
os.environ['PYTHON_EGG_CACHE'] = '/tmp/myuni/egg_cache'
 
root_dir = '/home/webapps/myuni/env'
 
# with mod_wsgi >= 2.4, this line will add this path in front of the python path
site.addsitedir(os.path.join(root_dir, 'lib/python2.6/site-packages'))
 
sys.path.append('/home/webapps/myuni/repository')
 
os.environ['DJANGO_SETTINGS_MODULE'] = 'myuni.settings'
 
import django.core.handlers.wsgi
 
application = django.core.handlers.wsgi.WSGIHandler()

The site.addsitedir() call in the WSGI file adds the library path in the virtual environment to the python path, by inspecting it to find all the packages that are installed. I then append the path to the project's repository to the path also, as is standard practice in this type of deployment.

The only other thing worth mentioning here, is the PYTHON_EGG_CACHE variable that I add to the environment. This is required if you are using Python modules that are packaged up as eggs, so that Apache can uncompress them in a directory to which it can write (/tmp/ on Ubuntu is 777 by default). If you are not using eggs, you can skip this line out altogether. Alternatively, you can uncompress the eggs first:

$ unzip [package].egg

You can read more about this issue, and the workaround, here.

Summary

That's pretty much all there is to it. This setup also makes it pretty easy to deploy new versions of my projects, for which I have started using Fabric. I'll save that for another post though.

If you do things differently, then I'd love to hear about how - so please leave a comment!

I didn't want to change the name of the project settings file to settings_local, however, as it would mean changing the WSGI file on every server that the project runs on.

The local_settings Trick

It's a well-known trick to use a file called local_settings.py or something similar, with a piece of code at the bottom of the main settings file:

try:
    from local_settings import *
except:
    pass

to override the value of settings variables. This will not work, however, if you wish to extend a settings variable (for example, adding an app to INSTALLED_APPS). For this, I have found that the following ugly hack seems to do the job.

The Ugly Hack

Replace the snippet at the bottom of the main settings file with the following code

try:
    LOCAL_SETTINGS
except NameError:
    try:
        from local_settings import *
    except ImportError:
        pass

What this is doing is effectively checking for the presence of a variable called LOCAL_SETTINGS. The local_settings file then contains this code:

LOCAL_SETTINGS = True
from settings import *

This means that the local_settings file will only be imported once, and it has all the variables in the main settings file available to extend at will. For example, to enable the Django debug toolbar:

MIDDLEWARE_CLASSES += ('debug_toolbar.middleware.DebugToolbarMiddleware',)
INTERNAL_IPS = ('127.0.0.1',)
INSTALLED_APPS += ('debug_toolbar',)

The code is ugly (and results in the main settings file being parsed twice) but it works!

The project has gone through several iterations since then, never really reaching any form of completeness. This time round, we are using proper version control (thanks to Mercurial and BitBucket), and trying to adhere to Django best practices as much as possible.

Straight from the repository website:

MyUni is a portal-style web application for Universities, offering students a central location to access all the relevant information about their course, modules, assignments, and staff members.

The project came about because of a passionate hatred for the WebCT system that our university is using, and a general feeling that things could be (and should be) better.

Django 1.2 will bring some really exciting new features, which I am sure would make developing and using MyUni a better experience. One of the things I'm most looking forward to using is the messages framework. I plan to use it for a bunch of things, possibly including form error messages...maybe.

Now, if only the release date would stay in the same place for more than a week!

To answer the questions for the compilers coursework, I wanted my sections to be numbered (1, 2, 3), my subsections to be numbered (a, b, c), and my subsubsections to be numbered (i, ii, iii). This numbering pattern matches that which is given in the question paper, whereby a number of nested enumerates were used. The default LaTeX section numbering however is (1, 1.1, 1.1.1). I finally found a way to alter this default behaviour by using the following code in the preamble:

 \renewcommand{\thesubsection}{(\alph{subsection})}
 
\renewcommand{\thesubsubsection}{\roman{subsection}.}

This code modifies the way that the counters for subsections and subsubsections are printed (section counters are correct when left at the default).

I hope this saves someone even the few minutes it took me to work this out, once I'd applied some logic! It's very useful to know that LaTeX has a counter for everything. This particular post was very helpful in working out which ones to alter, and how.

Domain authentication can be achieved relatively simply, by using either Winbind (part of the Samba project) or the new kid on the block, Likewise. Likewise-Open offers a simple procedure for joining domains, and the new version comes packaged with it's own version of Kerberos. It also hashes the Windows SID into a UNIX UID in a consistent manner - so the UIDs are always the same across your entire environment. Sounds perfect, right? Well, no. Not quite. The problem with Likewise-Open, is that it's difficult to integrate with Samba. Though it does ship with a "compatibility module" called lwi_compat, which allows Samba to hook into Likewise's authentication module, I found this quite difficult to get working, and I only achieved partial success through guesswork - as the documentation didn't actually help much, given they only officially support Samba 3.0.x (while Ubuntu now uses 3.3.x). When I did get it going, however, it only recognised the Windows users' primary group, not any of the other groups they were members of. This meant that my (possibly overly) complex system of ACLs and user directories just didn't work at all. So, on to Plan B...

When I read through the short (but sweet) Ubuntu Wiki article entitled ActiveDirectoryWinbindHowto, I felt like somewhat of a fool after reading a small, illusive section called Adding more than one Linux machine to a Windows network. Bingo! This section described a problem whereby the traditional Winbind domain authentication method would lead to inconsistent UIDs across the network, and thus cause headaches when trying to achieve anything like what I was aiming for. It suggested using a method of mapping SIDs to UIDs called RID. I assume this stands for Relative ID, which is another kind of ID Active Directory uses to track users within a domain. These can possibly clash from domain to domain, so it is advised not to use this method when your network contains a trust between multiple Windows domains, but for the simpler setup (like my own) it's a godsend.

This meant that I could use RID mapping within Winbind, which is part of Samba itself (so no troubles integrating those two), and achieve a consistent SID-UID mapping scheme across the network, allowing me to finally enable access to the file shares via. NFS. Amazingly, NFS "Just Worked" straight away, and I've written some nice wrapper programs around chown, getfacl and setfacl to set the correct owner and permissions on entire directory trees, which saves a lot of time when your UIDs are changing as often as mine were! I'm also using autofs to automatically map user's home directories on the Linux machines, which has proven itself to be very useful. I just used static fstab entries to map the other "general" file shares, like software and media - as I couldn't seem to get autofs direct maps working (apparently they are only partially working in Ubuntu anyway, but it seems as though they are completely broken to me).

On a side note, I've also just finished developing a Python-based rsync backup program, which allows me to write a _very_ small script to backup remote servers using rsync over SSH, and tar up the contents of all the servers into one archive. This is really useful, as I have a lot of disparate locations on different servers that all need to be pulled onto the backup drive every night. Now though, I should really concentrate on some revision for the exams I have after Christmas!

Happy holidays, everyone! (That's Merry Christmas and a happy new year, but just between you and me).

For example, when setting up Outlook Anywhere (the rebranded RPC/HTTP feature of Exchange, allowing Outlook users to connect from outside the organisation) I discovered a lot of "Sync Issues" appearing in my Inbox. The messages all had a common theme:

11:19:07 Synchronizer Version 12.0.6315
11:19:07 Synchronizing Mailbox 'User'
11:19:07 Synchronizing Hierarchy
11:19:07 Done
11:19:09 Microsoft Exchange offline address book
11:19:09              Not downloading Offline address book files.  A server (URL) could not be located.
11:19:09       0X8004010F

Clearly, something was wrong with the Offline Address Book. I was only getting these messages when using Outlook Anywhere, however, so this issue was obviously specific to RPC/HTTP.

Looking up the error code, I found that the problem I was experiencing was very common, but that nowhere seemed to have the ultimate repair. The information available was sparse, and I had to put together my own solution - which I will document below.

First, I registered an extra DNS (A) record for my email domain, called "autodiscover". I must be clear here, that this is for the mail domain, not for the domain used to access your OWA site. For example (and we'll go with the Microsoft classic here), if your users have addresses such as user1@contoso.com, user2@contoso.com and you access your OWA via https://mail.contoso.com/owa, then you need to register an A record for autodiscover.contoso.com.

Next, I prepared a new certificate request, that would hopefully end up with me obtaining a certificate that I could use to replace the current one, which would be valid for both mail.contoso.com and autodiscover.contoso.com (to continue with our example) - so that my Outlook clients could successfully access the autodiscover service, and download the OAB. To do this, I used the following EMS command:

New-ExchangeCertificate -domainname mail.contoso.com, exchange.contoso.local, autodiscover.contoso.com -Friendlyname "Contoso Exchange CAS SAN Certificate" -generaterequest:$true -keysize 1024 -path c:\certrequest.req -privatekeyexportable:$true –subjectname "c=GB o=contoso inc, CN=mail.contoso.com"

This command requires a little explanation. The -domainname switch is used to specify a list of addresses for which this server is valid. This is called a SAN (Subject Alternative Name). Not all CA's support SANs, but Windows Server 2008's CA Services does, which I will come back to later. Next, we give the certificate a "Friendly Name", which is just a reference for you, the administrator. Then we specify that we are looking to generate and save a request, and that we want to be able to export the private key. The Subject Name is important, but also slightly confusing. You must specify your country code (US, GB, ES), your organisation name, and Common Name (CN) - which is the most important one. This must be the URL used to access the SSL service using a web browser, so mine was mail.contoso.com.

Once this request is saved, I passed it on to my CA to get the certificate issued. If you're using a 3rd party CA (like VeriSign), then you'll have to check first whether they support SANs. I use self-signed certificates, and my CA is running Windows Server 2008, which does support SANs, so I issued the request internally. This is done by accessing the CertSrv website, at http://servername/certsrv, and clicking the "Request a Certificate" link. Then, I chose "Advanced Request", and pasted the reqest file's contents into the box, and picked the "Web Server" template.

This presented me with a downloadable certificate, which I saved locally in CER format on the exchange server. Then I used the following command to import the certificate:

Import-ExchangeCertificate –path <certificate>

Once the certificate was imported, I enabled it for use with exchange. A similar command is used for this:

Enable-ExchangeCertificate

This prompted for a list of services, where I entered IMAP, IIS, SMTP as these are the default installed services. Only IIS actually gets used here, so I shouldn't worry too much about this one. If you're not sure, then just enter the same as me. Lastly, it asked for a thumbprint, which I copied and pasted from the output of the import command. Finally, after accepting the confirmation, the certificate was enabled.

And that was it. Both OWA and Outlook Anywhere are now working perfectly, and hopefully this post will help at least one other lost soul with the same problem!

Something strange happens to me every time exams come around. I seem to pick up new projects, and just run with them. This time, I've become involved with a small group of people at university, writing a portal-style information system for universities. I suppose most people call this behavior procrastination, but I'm quite deeply in denial about that.

Ever since my post about the Backtrac Backup System, I've been really enjoying using Django. Something about it just makes developing for the web, well, exciting. That can only be good, right? I am the designated server administrator for this latest project, mostly due to the fact that I am the only one with a server to administer, and some of the things I've learned so far seem worthy of a mention here.  Firstly, we as developers were - how can I put it - stepping on each other toes somewhat. The project at this point had no version control, so we were just editing a bunch of files over SFTP. Obviously, some sort of Source Control Management was in order. I did some research, and decided that Git was a nice, modern alternative to the ever-popular SVN. It also meant that my server was constantly backed up by everyone on the team - but that's just a bonus!

So, I installed Git, and started a repository. A lot of effort went into learning how the system works, and more importantly, how to make it work for us. Directed Acyclic Graphs thankfully made some sense to me, so I could just about understance the documentation. I wrote some custom hooks, and a C Program to syncronise the web-server. I was happy, and absolutely certain that this was the solution to all our woes. I was mistaken. Git just didn't work the way I had hoped. The custom hooks were throwing permission errors all over the place, and my development team (read: my friend Rob Miles) was locked out of the repository. We made the decision yesterday to scrap Git, and go back to the previous system of editing the files over SFTP. We are always in constant communication when developing for the project, so it's not too big a deal, but I feel that I failed as an adminisrator. You see, as an admin your job isn't just to play with cool toys and loud servers - your primary purpose is to give the users what they need to work, and that is most certainly not what I achieved.

I'm glad I took the time to learn Git - and I'm sure it will help me later in life, in some way or another. It's just unfortunate that it didn't work out the way I had hoped for our project.

The basic idea is that Firefox has a file called profiles.ini, which takes care of all the configured profiles, and where they are stored. I used this file to change the default profile location to within the user's home directory on the file server. I had to use a home drive, mapped to the root of my users' folder redirection directory on the server, as I presumed UNC paths were unsupported in the .ini file. I did this with a GP Preference drive map using the %USERNAME% variable, and an amazing feature of GP Preferences - parsing and even altering ini files. You can specify which section of the ini file you are interested in, and which key you want changing. How useful! My policy looks like this:

You can probably work out what's going on here, but I'll give a quick overview. Basically, you specify the ini file to edit - in this case it's %APPDATA%\Mozilla\Firefox\profiles.ini. Using the %APPDATA% variable means that it will always resolve to the correct location in the user's local profile, whether they are on XP or Vista. Then we specify the section of the ini file - I'm interested in Profile0. This is the only profile present by default, but allows users to have multiple profiles if they wish without overwiting their settings when they log off. Finally, you specify the key to change, and what to change it to. I rename the default profile to Firefox, and change it's location to H:\Firefox. Also, this path is not relative, so I have to change IsRelative to 0.

OK, so now we have the Firefox profile location sorted, we need to make sure that folder exists - or Firefox will just overwrite our changes and make it's own folder in the default location. It's easy to use GP Preferences for this as well - as there's a Folders preference category. So I just made a new folders preference for \\zeus\UserData\%USERNAME%\Firefox, with the action of create (zeus is my main DC and File Server). I used the UNC path to be sure that the folder is created, even if the drive map hadn't come into effect when this preference was applied. Also, a little trick I had to pull here was ticking the checkbox titled Run in logged-on user's security context on the common tab. This is because only the user has permissions on their home directory, so this preference needed to run in the context of that user for it to work successfully (without access denied errors).

Once this was finished, the system started to work flawlessly. I copied the contents of existing Firefox profiles to the newly created directories, and they were picked up by Firefox with no problems. New users get blank profiles as expected, but they are stored on the file server instead of the local machine. One little issue I have encountered is that a user can't logon at more than one machine, and start Firefox - as the program can't lock particular files in the profile. This just results in a message saying this Firefox is already running though, which is pretty much correct (and I can't see why this would ever cause problems for the user).

The last trick I employed, to make things a little speedier and to reduce uneccesary file server traffic, was to disable disk caching on the roaming Firefox profiles. To do this, I used a file policy in GP Preferences to copy a tiny user.js (Firefox's preference file override) which contained only one line:

user_pref("browser.cache.disk.enable", false);

This turns off disk caching completely, which will not only save space on the file server, but should speed things up as well. I hosted this file elsewhere on the file server, and told the file policy to simply copy it into place, within the user's Firefox profile. Here's the preference:

So there you have it, my technique for enabling roaming Firefox profiles. If you've achieved the same through a different method, or have any ideas on this this could be improved, then I'd love to hear how - feel free to comment on this post.