• The Django server (yet to be named)
• The client daemon (backtracd)
• The backup library itself (backuplib)

As I mentioned in the previous post, the Django server will be based around Celery. This will offer me the ability to schedule a task for a certain date and time in the future, which is a huge part of the implementation taken care of.

One thing I am yet to figure out, however, is how the server will communicate with the client daemons. At the moment I'm thinking XML-RPC, but it seems to me that it will end up a sort of mish-mash of different technologies - as the Django server will be exposing a JSON API (probably via Piston). I have also considered running Django on the clients too, so everything can use JSON/Piston, but this seems at the moment like an unnecessary burden.

Another thing to consider is the data that is to be transmitted between the nodes. Should I use a bunch of fields in a JSON object to represent the success/failure of a job, and how much data was backed up, etc.? Another option is to pickle a Result object of some sort - representing a result as a Python instance to be interpreted by the server. These are the things I love to think about when designing a system, and I'll certainly have plenty to write about in the inevitable report.

Something I can work on in the meantime, however, is backuplib - the client library that will actually copy the data. At the moment, it's just a wrapper around rsync/tar. That probably won't change, but it will need to get a bit smarter in the way it handles failures (and restores, of course) if it's going to make for the basis a decent backup system.

Backtrac is a pretty old project, and one whose name I am likely to change once I'm working on it. It was started in my first semester of university last year, when I didn't have enough work to keep me busy. I didn't know enough about Django back then, though, so it was left unfinished and only half-working.

Now I think I know enough to pull this off, and it's a chance to do something I really enjoy - while getting marks for it in the process! If everything goes to plan I'll be supervised by Dr. Julie Greensmith, which I'm really looking forward to.

I'll be using an app called Celery to handle the background tasks, which will be simple "pings" to the client machines, telling them to start a backup. Celery uses AMQP, provided by a broker such as RabbitMQ (which is written in Erlang). It's usually used for processing intensive computational tasks, but can also be put to use for jobs that need executing on a periodic basis, like backups.

I can tell you're getting too excited, so I'll stop. Rest assured, though, I will be writing about the discoveries I make along the way to finishing this ambitious project.

Beginnings

Firstly, I use virtualenv to keep the Python environments for the different projects separate, but that's a topic in itself. This isn't a requirement, but it helps if you have dependencies that you don't want installed system-wide, or projects that use different versions of a particular package.

You can read more about virtualenv and Django here.

The directory structure for my webapps directory (which lives inside /home/) looks something like this:

webapps/
|-- apache
|   |-- coral.conf
|   |-- coral.wsgi
|   |-- myuni.conf
|   `-- myuni.wsgi
|-- coral
|   |-- env
|   `-- repository
|-- myuni
|   |-- env
|   `-- repository

The two projects shown here are Coral Issue Tracker and MyUni. The directory for each project contains the virtual env for that project, and the repository. Alongside the project folders is a directory called apache, which holds two files for each project: the WSGI file (<project-name>.wsgi) and the Apache configuration file (<project-name>.conf).

Apache Configuration

Then, in the main Apache configuration, I use a directive that includes all these files automatically:

Include    /home/webapps/apache/*.conf

The individual project configuration files then contain a set of simple directives almost straight from the Django deployment documentation. For example, the MyUni configuration looks like this:

WSGIScriptAlias  /myuni        /home/webapps/apache/myuni.wsgi
Alias            /myuni/docs   /home/webapps/myuni/repository/docs/_build/html
Alias            /myuni/media  /home/webapps/myuni/repository/myuni/static

WSGI Files

The WSGI files are also pretty simple, though there are a couple of things worth explaining. Here is the WSGI file for the MyUni project:

import os
import sys
import site
 
os.environ['PYTHON_EGG_CACHE'] = '/tmp/myuni/egg_cache'
 
root_dir = '/home/webapps/myuni/env'
 
# with mod_wsgi >= 2.4, this line will add this path in front of the python path
site.addsitedir(os.path.join(root_dir, 'lib/python2.6/site-packages'))
 
sys.path.append('/home/webapps/myuni/repository')
 
os.environ['DJANGO_SETTINGS_MODULE'] = 'myuni.settings'
 
import django.core.handlers.wsgi
 
application = django.core.handlers.wsgi.WSGIHandler()

The site.addsitedir() call in the WSGI file adds the library path in the virtual environment to the python path, by inspecting it to find all the packages that are installed. I then append the path to the project's repository to the path also, as is standard practice in this type of deployment.

The only other thing worth mentioning here, is the PYTHON_EGG_CACHE variable that I add to the environment. This is required if you are using Python modules that are packaged up as eggs, so that Apache can uncompress them in a directory to which it can write (/tmp/ on Ubuntu is 777 by default). If you are not using eggs, you can skip this line out altogether. Alternatively, you can uncompress the eggs first:

$ unzip [package].egg

You can read more about this issue, and the workaround, here.

Summary

That's pretty much all there is to it. This setup also makes it pretty easy to deploy new versions of my projects, for which I have started using Fabric. I'll save that for another post though.

If you do things differently, then I'd love to hear about how - so please leave a comment!

The project has gone through several iterations since then, never really reaching any form of completeness. This time round, we are using proper version control (thanks to Mercurial and BitBucket), and trying to adhere to Django best practices as much as possible.

Straight from the repository website:

MyUni is a portal-style web application for Universities, offering students a central location to access all the relevant information about their course, modules, assignments, and staff members.

The project came about because of a passionate hatred for the WebCT system that our university is using, and a general feeling that things could be (and should be) better.

Django 1.2 will bring some really exciting new features, which I am sure would make developing and using MyUni a better experience. One of the things I'm most looking forward to using is the messages framework. I plan to use it for a bunch of things, possibly including form error messages...maybe.

Now, if only the release date would stay in the same place for more than a week!

So, I have been working on a totally new design, of my own creation this time. The main site has already been changed over, and I'm working on themeing the Wordpress install as we speak.

To take a look at the new theme's progress, check it out here. The new theme is now live on the site, I hope you like it!

I'll be sure to update you when it's complete. In the meantime however, feedback, as always, is most welcome.

Update: The new design is now live! I'll still be working on it though, fixing up any little niggles that are bound to crop up.

To answer the questions for the compilers coursework, I wanted my sections to be numbered (1, 2, 3), my subsections to be numbered (a, b, c), and my subsubsections to be numbered (i, ii, iii). This numbering pattern matches that which is given in the question paper, whereby a number of nested enumerates were used. The default LaTeX section numbering however is (1, 1.1, 1.1.1). I finally found a way to alter this default behaviour by using the following code in the preamble:

 \renewcommand{\thesubsection}{(\alph{subsection})}
 
\renewcommand{\thesubsubsection}{\roman{subsection}.}

This code modifies the way that the counters for subsections and subsubsections are printed (section counters are correct when left at the default).

I hope this saves someone even the few minutes it took me to work this out, once I'd applied some logic! It's very useful to know that LaTeX has a counter for everything. This particular post was very helpful in working out which ones to alter, and how.

I've been working with Django more and more lately, and I've written my first "commercial" application using my new favourite framework - a booking system for taught causes at the City Council. I'm really enjoying writing web applications with Django, and I'm sure this blog will start to resemble a web-developer's in the near future.

Also, I'm really looking forward to this year at university. As I understand it, there's a lot more work to be done, but the software engineering group project should be fun - as long as my "randomly chosen" team are happy with us using Python!

For example, when setting up Outlook Anywhere (the rebranded RPC/HTTP feature of Exchange, allowing Outlook users to connect from outside the organisation) I discovered a lot of "Sync Issues" appearing in my Inbox. The messages all had a common theme:

11:19:07 Synchronizer Version 12.0.6315
11:19:07 Synchronizing Mailbox 'User'
11:19:07 Synchronizing Hierarchy
11:19:07 Done
11:19:09 Microsoft Exchange offline address book
11:19:09              Not downloading Offline address book files.  A server (URL) could not be located.
11:19:09       0X8004010F

Clearly, something was wrong with the Offline Address Book. I was only getting these messages when using Outlook Anywhere, however, so this issue was obviously specific to RPC/HTTP.

Looking up the error code, I found that the problem I was experiencing was very common, but that nowhere seemed to have the ultimate repair. The information available was sparse, and I had to put together my own solution - which I will document below.

First, I registered an extra DNS (A) record for my email domain, called "autodiscover". I must be clear here, that this is for the mail domain, not for the domain used to access your OWA site. For example (and we'll go with the Microsoft classic here), if your users have addresses such as user1@contoso.com, user2@contoso.com and you access your OWA via https://mail.contoso.com/owa, then you need to register an A record for autodiscover.contoso.com.

Next, I prepared a new certificate request, that would hopefully end up with me obtaining a certificate that I could use to replace the current one, which would be valid for both mail.contoso.com and autodiscover.contoso.com (to continue with our example) - so that my Outlook clients could successfully access the autodiscover service, and download the OAB. To do this, I used the following EMS command:

New-ExchangeCertificate -domainname mail.contoso.com, exchange.contoso.local, autodiscover.contoso.com -Friendlyname "Contoso Exchange CAS SAN Certificate" -generaterequest:$true -keysize 1024 -path c:\certrequest.req -privatekeyexportable:$true –subjectname "c=GB o=contoso inc, CN=mail.contoso.com"

This command requires a little explanation. The -domainname switch is used to specify a list of addresses for which this server is valid. This is called a SAN (Subject Alternative Name). Not all CA's support SANs, but Windows Server 2008's CA Services does, which I will come back to later. Next, we give the certificate a "Friendly Name", which is just a reference for you, the administrator. Then we specify that we are looking to generate and save a request, and that we want to be able to export the private key. The Subject Name is important, but also slightly confusing. You must specify your country code (US, GB, ES), your organisation name, and Common Name (CN) - which is the most important one. This must be the URL used to access the SSL service using a web browser, so mine was mail.contoso.com.

Once this request is saved, I passed it on to my CA to get the certificate issued. If you're using a 3rd party CA (like VeriSign), then you'll have to check first whether they support SANs. I use self-signed certificates, and my CA is running Windows Server 2008, which does support SANs, so I issued the request internally. This is done by accessing the CertSrv website, at http://servername/certsrv, and clicking the "Request a Certificate" link. Then, I chose "Advanced Request", and pasted the reqest file's contents into the box, and picked the "Web Server" template.

This presented me with a downloadable certificate, which I saved locally in CER format on the exchange server. Then I used the following command to import the certificate:

Import-ExchangeCertificate –path <certificate>

Once the certificate was imported, I enabled it for use with exchange. A similar command is used for this:

Enable-ExchangeCertificate

This prompted for a list of services, where I entered IMAP, IIS, SMTP as these are the default installed services. Only IIS actually gets used here, so I shouldn't worry too much about this one. If you're not sure, then just enter the same as me. Lastly, it asked for a thumbprint, which I copied and pasted from the output of the import command. Finally, after accepting the confirmation, the certificate was enabled.

And that was it. Both OWA and Outlook Anywhere are now working perfectly, and hopefully this post will help at least one other lost soul with the same problem!

Something strange happens to me every time exams come around. I seem to pick up new projects, and just run with them. This time, I've become involved with a small group of people at university, writing a portal-style information system for universities. I suppose most people call this behavior procrastination, but I'm quite deeply in denial about that.

Ever since my post about the Backtrac Backup System, I've been really enjoying using Django. Something about it just makes developing for the web, well, exciting. That can only be good, right? I am the designated server administrator for this latest project, mostly due to the fact that I am the only one with a server to administer, and some of the things I've learned so far seem worthy of a mention here.  Firstly, we as developers were - how can I put it - stepping on each other toes somewhat. The project at this point had no version control, so we were just editing a bunch of files over SFTP. Obviously, some sort of Source Control Management was in order. I did some research, and decided that Git was a nice, modern alternative to the ever-popular SVN. It also meant that my server was constantly backed up by everyone on the team - but that's just a bonus!

So, I installed Git, and started a repository. A lot of effort went into learning how the system works, and more importantly, how to make it work for us. Directed Acyclic Graphs thankfully made some sense to me, so I could just about understance the documentation. I wrote some custom hooks, and a C Program to syncronise the web-server. I was happy, and absolutely certain that this was the solution to all our woes. I was mistaken. Git just didn't work the way I had hoped. The custom hooks were throwing permission errors all over the place, and my development team (read: my friend Rob Miles) was locked out of the repository. We made the decision yesterday to scrap Git, and go back to the previous system of editing the files over SFTP. We are always in constant communication when developing for the project, so it's not too big a deal, but I feel that I failed as an adminisrator. You see, as an admin your job isn't just to play with cool toys and loud servers - your primary purpose is to give the users what they need to work, and that is most certainly not what I achieved.

I'm glad I took the time to learn Git - and I'm sure it will help me later in life, in some way or another. It's just unfortunate that it didn't work out the way I had hoped for our project.

• Pyhon in general
• XML-RPC
• Filesystem hardlinks
• The MVC concept, and Django

This should be enough content to fill a 15-minute slot, I hope.

Now I'll explain a little bit about how BackTrac works, for the benefit of those not coming to the presentation.

What I wanted to achieve with BackTrac was a smart backup system, that doesn't necessarily have to be the fastest. I wanted a very detailed web interface, with scheduling capabilities and log view. I decided on Django for this, seeing as I had already decided on Python for the system itself.

The nodes in the system use XML-RPC to communicate, and SMB for the actual file-copying. The real bonus of this system is that it takes advantage of a little-understood feature, present in most modern filesystems, called hardlinks. Hardlinks enable the user to essentially point to the same piece of data on a disk from more than one position. Essentially the concept is that if a file has been backed up before, why back it up again? Just create a link or "shortcut" (a hardlink) to the previously backed-up file. This is the basis for BackTrac.

Also, Django is turning out to be a real treat. This is the first time I have used an MVC like Django, and I'm really enjoying it. Web development is exciting again! I do agree with one point however, that Django makes the easy things easy, and the hard things impossible. Not quite impossible in my case, but I've had to do some pretty strange things to get Django to do what I want.

Finally, I've had the BackTrac project approved on Sourceforge.net, so watch out - I'll be doing the initial import soon and getting some web content online. First though, I have to decide on the best way to distribute the application. This is a bit more complicated than it might otherwise be, because there are three different aspects of the system - the client application, the server application, and the Django project. On this topic or any other, as usual, opinions are most welcome.