Investigating Configuration

But Where Does It Come From?

Have you ever wondered where the configuration for CruiseControl.Net comes from? How about what happens to convert from the configuration to the projects list? And then, the question that I’m interested in, how can we extend it?

As these questions are all tied together, I’m going to run a series on posts on answering them. Why? I’m looking at adding security to CruiseControl.Net, and any security system needs to store its configuration information somewhere. So rather than write my own configuration system for the security, I figured I’d just extend the existing configuration system. Along the way I might take a couple of detours to address other issues, but it’ll all be focused around the configuration.

A Starting Point

My first question addressed the issue of where the configuration is stored, and it has an easy answer – ccnet.config. This is a file that is stored in the file system (the location can be configured). It’s an XML file in a similar vein to the configuration files for a .Net application. I can think of some reasons why it was done this way:

• Ease of set-up: no need to install or configure a database server or other storage systems.
• Portability: can easily just copy it from one machine to other.
• System-independence: every operating system has a file system, therefore no need to worry about converting for Linux, etc.
• Easily extended: can add extensions without needing to add additional configuration processing

Having said that, the way the configuration loading works we could write a configuration loader that would handle an alternate storage type (e.g. a database). The configuration is implemented using a series of interfaces – of which one or more can be replied (you’d have to add code to wire them up though). These classes are:

• IConfigurationService: service interface for exposing the configuration system
• IConfigurationFileLoader: interface for loading the configuration settings
• IConfigurationFileSaver: interface for saving the configuration settings
• IConfiguration: interface to expose the actual configuration settings

IConfigurationService only uses IConfiguration, it know nothing about the other two interfaces. There are three implementations, of which only one does know about these interfaces. The implementations are:

• FileConfigurationService: an implementation that uses IConfigurationFileLoader and IConfigurationFileSaver to load and save from the file system.
• FileWatcherConfigurationService: an implementation that monitors any file changes and if detected tells any listeners to reload. This uses an IConfigurationService instance internally to actualy load the settings.
• CachingConfigurationService: the settings are cached so the system doesn’t need to go to the file system everytime the settings are queried. This also uses an IConfigurationService instance internally to actualy load the settings.

In the current code these three services are chained. That is CachingConfigurationService uses an instance of FileWatcherConfigurationService, which in turn uses an instance of FileConfigurationService (the FileWatcherConfigurationService instance can be skipped). If we wanted to store our settings in a database we would write a database service and then change the start-up process to use this instead of one of the three existing implementations (choose which other functionality is required).

Loading from the File

The actual loading from the file is handled by an instance of IConfigurationFileLoader (assuming we’re using FileConfigurationService). The default implementation, DefaultConfigurationFileLoader, uses the Exortech.NetReflector library to handle the XML deserialisation. If we wanted to use another file-system format (say CSV), we could write our own implementation of IConfigurationFileLoader to handle this and we wouldn’t need to change anything else! Pretty neat in my opinion

Back to DefaultConfigurationFileLoader. It uses another class called NetReflectorConfigurationReader, to call into the Exortech.NetReflector library.This is mainly a wrapper class to handle loading each project and adding it to the projects list in an IConfiguration instance.

The Exortech.NetReflector library library appears to do some cool stuff around dynamically loading types from some attributes. If we take a look through some of the various classes that are loaded from configuration we see instances of ReflectorType and ReflectorProperty attributes. These obviously tell Exortech.NetReflector how to handle the XML data and which classes to generate. I don’t know too much about them, so I’m not going to go into detail here.

Extending the Configuration

So far we covered my first two questions, that merely leaves how to extend the configuration. If we want to add a new instance of an already existing interface (e.g. an ITrigger, ITask, IProject, etc.) this is easy. We just add the new class, make sure it implements the correct interface and then decorate it with ReflectorType and ReflectorProperty attributes.

We use the ReflectorType attribute to tell the loader which class instance to instantiate. Then, each property that we want to load we decorate with a ReflectorProperty attribute. And that’s all there is to it. When we add our new configuration into CCNet.config the loader will handle it for us automatically. Sure makes extending the existing interfaces simple.

Coming Up

I’ve covered very quickly how to add new implementations of existing interfaces. However for the security sub-system I’m thinking of adding an entirely different type of interface – completely unrelated to what’s already there. Not only will it be new, it will need to hook into a couple of different places. Therefore I’ll need to do more than just decorating my classes with ReflectorType and ReflectorProperty attributes. In my next post in this series I’m planning on looking at just what we need to do for this scenario.

An Introduction to Queues

What are Queues?

I’ve noticed in the mailing lists that there have been a couple of questions about queues lately – either wanting to modify their functionality or thinking there is an error in the way they work. So I took a quick look into the code to try and understand how they work. This posting is my understanding of their current functionality.

To answer my question, a queue is a group of projects. It acts in a FIFO (first-in, first-out) manner to control which projects get built. The purpose behind a queue is only one project in the queue will be built at any one time – hence the FIFO requirement. It also handles prioritisation and prevents duplicates.

But How Do They Work?

You can think of a queue as a wrapper around a project. When CruiseServer recieves a request to build a project (either from a user or from a trigger) it actually calls the queue to schedule the build. If there are no other projects in the queue, then the build gets started immediately, otherwise it has to wait to come to the top of the queue.

The actual queue functionality is implemented in a class called IntegrationQueue. IntegrationQueue is basically an array with some smarts around how items get added. The two main methods for this are Enqueue() and Dequeue(). Enqueue() checks to see if the project is already in the queue, if it is not it checks to see its priority and then adds it to the queue in the correct position. Dequeue() removes the project at the head of the queue and notifies its integrator that it can start building.

And that’s really all there is to queues (there are some other methods around finding out if there are queued items, removing unwanted project builds, etc. but these are ancillary functions).

Missing Builds

A common issue that has been raised on the mailing lists lately around conflicts in builds. We have the following scenario:

• Project A depends on Project B – so a ForceBuild trigger has been added to Project A.
• Project A also has an interval trigger with an IfModificationExists.
• Project B uses an interval trigger to build

The following happens:

1. Project B starts its build
2. Project A is trigger (interval trigger), gets added to the queue with IfModificationExists
3. Project B completes its build, Project A detects this
4. Project A attempts to add itself to the queue with ForceBuild
5. The queue detects it is a duplicate and discards the request
6. Project A starts its build using IfModificationExists
7. …

So, we see the problem is not with how the queues work, but with adding the result from the ForceBuild trigger. As far as I’m aware there is no work-around for this issue. Probably the best approach would be to modify the trigger so it overrides any existing queued project build. If I get some time I’ll look into it (or someone else can give it a try ).

Locking Queues

Another scenario comes when you have multiple queues – queues A, B & C, each with multiple projects. Queue A could be the application framework, while queues B & C are applications. If Queue A is building, then we don’t want to try and build the applications, otherwise we’d get conflicts.

Again, this scenario isn’t handled by the current queue mechanism. Queues are pretty much silos – they don’t communicate with each other. To handle this scenario we’d need to add some way of communicating between the queues (perhaps a lock object). When a queue wants to schedule a build it would need to check any dependant queues for a lock. If there are any locks, then it would need to wait. If there are no locks, then it would lock itself, build the queued projects and then release the lock. At this stage it would then need to notify any dependant queues that they can start.

That’s All Folks

That covers how queues work, and some of the current issues around queues. They are reasonably simple beasts, we just have to understand their limitations. Of course, over time they’ll probably increase in functionality (and complexity) as people refine them, but for now they are a simple, elegant approach to a problem.