Inner Workings: The Project Scheduler

The Heart of the Matter

The core of CruiseControl.Net is the project scheduler. This is the piece of code that is responsible for scheduling builds – without which CruiseControl.Net just wouldn’t work.

Before I delve into the actual workings of the scheduler, let’s quickly review how builds can be scheduled.

First and foremost, in order to schedule a project build a trigger is required. There are a number of different types of trigger – from interval triggers to scheduled triggers to projects that monitor other locations or projects. Additionally triggers can be combined or filtered. But one thing all filters have in common is they tell the scheduler a build needs to be performed.

The second part of the scheduler is the queues. Early versions of CruiseControl.Net had each project running in its own little world – they didn’t affect other projects. As the number of projects increased, this lead to increased contention on the build servers and a lack of resources. To counter this queues were added. A queue is a group of projects, of which only one can have a running build at any point in time. Generally they work on a first-in, first-out basis, but it is possible to set queue priorities.

With this background, let’s delve into how project scheduling actually works

Managers, Integrators and Queues

The main class that handles everything in CruiseControl.Net is CruiseServer. This is responsible for starting everything and handling all user interactions. But, it doesn’t actually handle the scheduling of builds. This is handled by a number of other classes.

Looking at CruiseServer, there is a IntegrationQueueManager class. This encapsulates all the actual projects and their integrators. Now an integrator implements IProjectIntegrator and is responsible for the actual triggering of a build. In a moment I’ll return to how it does this, but first, how is an integrator started.

When IntegrationQueueManager is instantiated it iterates through all the projects and ensures that there is a queue for each project. If there is no queue, then it creates a new queue with the same name as the project. These queues are all added to an IntegrationQueueSet.

Once the queues have been initialised, the IntegrationQueueManager then calls ProjectIntegratorListFactory to generate all the project integrators. As well as containing the project configuration, the integrator also contains a reference to the associated queue. At the moment there is only one IProjectIntegrator – ProjectIntegrator.

This completes the initial setup of the queues and integrators, the next step is to start the projects integrating. This is done by calling the StartAllProjects() method (by CruiseServer), or by calling Start() for a specific project (StopAllProjects() and Stop() do the opposite).

When StartAllProjects() is called, it iterates through all the project integrators and checks to see whether the project can start. This involves checking the configuration and then the state persistence (both new in 1.4.3). If both these checks start, then the integrator is started. The actual starting of the integrator is done by calling the Start() method on the integrator.

Nice and simple, but here’s a diagram to illustrate this process:

Triggering Builds

The above initialisation got to the point of calling Start() on ProjectIntegrator. This method starts a new thread that contains a polling loop. This loop checks to see if there is an integration every 100ms. If there is an integration it then calls the Integrate() method on Project which performs the actual integration (e.g. pre-build, source control, tasks and publishers).

This check consists of two parts. First it checks the queue to see if there is a pending integration request. If there is a request, it locks any queues that need to be locked, starts a new request and calls the Integrate() method. After this it cleans up and exits the check logic.

The second part of the check, which is only performed if there is no pending request, is to check all the triggers. Each trigger has a Fire() method, which performs the actual check. The output of the fire method is an integration request or null – if the output is not null then it gets added to the queue.

The root level trigger is a combination trigger, which merely iterates through each child trigger and calls its Fire() method – it is up to each child trigger whether it returns a request or not.

Once the trigger checking has finished, it checks the queue to see if the request is next. If not, it enters a loop until the request is ready. Then when the request is ready, the check logic finishes – it doesn’t actually call Integrate() after the trigger checking. What happens is the polling loop goes through another cycle and then the integration request gets returned from the first part of the check.

The following diagram shows this:

Queues

The final piece of the puzzle is the integration queues. I’ve already mentioned them a couple of times, but let’s pull them apart and see how they actually work.

First of all, queues do not use the built-in queue classes – they use a List<> instance instead. The reason for this very simple – they do more than just adding and removing items – they also allow re-ordering (based on priorities). Plus items on the queue are not removed until completed – which would cause issues with de-queuing.

In turns of how they work. When an item comes in, the queue checks to see if it already exists. If the item exists it applies any re-ordering rules (ignore, re-add or replace existing), otherwise it just adds it to the queue. It will look at any other items in the queue and then add it after the last item with the same priority, or before the next highest priority.

When the integrator checks for a request, it will always return the item in position 0 (the start of the list). This item remains there until the integrator performs its clean-up.

Finally, there are a few call-back methods that are used to synchronise the state between the queue and the integrator.

And that’s all there really is to queues – very, very simple. The following diagram shows how queues relate to the integration polling cycle.

Summary

This post has covered how project integrations work and how builds are scheduled.

The main driver for the process is IntegrationQueueManager, which responsible for initialising everything and then starting the actual integration cycle. The actual integration cycles are handled by a polling thread within each integrator.

The actual process for starting a build is controlled by both the integrator and the queue. The integrator checks the triggers to see if a build should be scheduled, and when one is found it adds it to its associated queue. Then, in the next polling cycle it retrieves the request off the queue and actually performs the integration.

The queues act to limit the number of project builds at any time, and do this by storing requests. The integrator will only perform a request when it is the first item in the queue.

Final Note

This post is about how builds are scheduled – I haven’t covered how an actual build is performed. But that is an entirely different topic, so I will leave that for another time