CruiseControl.NET – vNext – TaskExecutionContext (cont.)

In my last post I looked at how the new TaskExecutionContext works at a high level. In this post I’ll describe the current methods on the context and where they fit into the process.

When an integration is started a new TaskExecutionContext is created using StartNew() on ITaskExecutionFactory. This creates a new folder for the integration, starts an XmlWriter for the build log and adds the basic project details to the log.

The next call in the sequence is to StartChild() on TaskExecutionContext. This creates a new child instance of the context that is specific to a task. This will output the task details (e.g. name and type) to the log. An instance is created for each task by the Project – this then gets passed into the task.

Once a task has finished the Complete() method is called. This writes the final details of the task to the log and tidies up. The same complete method is called whether the context is for a task or for an integration. The only difference is the integration level call will close the log at the same time.

There are currently four methods that can be called from within a task:

• AddEntryToBuildLog(): adds a comment directly to the log
• ImportFile(): copies/moves a file to within the folder for the integration and adds an index entry to the log
• StartOutputStream(): creates a new file within the integration folder and adds an index entry to the log
• AddModifications(): adds any modifications from a source control block to the context

There are also a number of properties that are available. These are:

• CurrentStatus: the current status of a task. This can be set by a task but if not set then calling Complete() will set it to Success.
• IsCompleted: a read-only property flag indicting whether a context has been completed. Calling any methods on a completed context will throw an exception.
• Request: the original integration request details (read-only).
• Project: the project the integration is for (read-only).
• Parent: the parent context. If the context is the project level context this will be null. Otherwise this will point to the context that created the current context (i.e. the context that StartChild() was called on)(read-only).
• ModificationSets: the current modifications within the integration (read-only).

One point to note about ModificationSets is there is only a single instance for an integration. This is why AddModifications() is required – it handles any locking required so they are no thread conflicts when adding or retrieving modifications.

So despite the changes to how the context works, from a task authoring perspective it should be even simpler to use.

CruiseControl.NET – vNext – TaskExecutionContext

One of the important areas in vNext is the TaskExecutionContext. This is both taking over from IIntegrationResult in the current versions and providing additional functionality.

One of the major differences between TaskExecutionContext and IIntegrationResult is there will be a single TaskExecutionContext per task. In the current versions there is only a single IIntegrationResult for an integration – it is up to the tasks themselves to correctly manage the properties on the instance. As you can imagine this can potentially led to issues where the properties are treated differently between the different tasks. So in vNext each task will have its own TaskExecutionContext instance. The controller (typically the project) is responsible for starting these instances and completing them once the task has completed.

This important difference also affects how the contexts are started. The initial context (at the project level) is started via a factory class (TaskExecutionFactory). This will initialise all the required settings for the instance. Subsequence contexts are then started via the StartNew() method on the current context. This generates a new context that is associated with a parent – which means the log entries will be written to the same log.

This is important because of a second major change – log entries are now written directly to a file instead of being stored in memory. This is to get around the nasty OutOfMemory problem that we are having with the current versions. As such this significantly changes the way that the build log is being written. In the current versions the log is written by (and only by) an xmlLogger task. If this task is omitted or fails to execute then there will be no log. As you can imagine this can be slightly disconcerting for new users (I remember it was for me!) Now because the log is emitted straight to disk it will always be written – which makes the xmlLogger task obsolete.

With this in mind, the format of the file has changed significantly. I am still working on the format and re-adding all of the goodness that’s in the current versions, but here is an example of what the file is shaping up as:

<?xml version="1.0" encoding="utf-8"?>
<project name="TestProject">
  <start>2011-01-09T15:28:43</start>
  <task type="Comment">
    <start>2011-01-09T15:28:44</start>
    <entry time="2011-01-09T15:28:44">A Test Comment</entry>
    <finish>2011-01-09T15:28:44</finish>
    <status>Success</status>
  </task>
  <task type="MergeFiles" Name="NUnitFiles">
    <start>2011-01-09T15:28:44</start>
    <file time="2011-01-09T15:28:44">testproject.xml</file>
    <file time="2011-01-09T15:28:52">otherproject.xml</file>
    <finish>2011-01-09T15:28:53</finish>
    <status>Success</status>
  </task>
  <finish>2011-01-09T15:28:53</finish>
  <status>Success</status>
</project>

This now contains a wealth of information on the actual tasks that were executed. Details for each task appears within the tasks themselves (this will include references to imported files, embedded data, etc.) We can quickly and easily see:

• The name (if any) and type of a task
• The start and finish times
• The final status
• What data was added during the task

I’ll also flesh out the project data to include most of the data that is in the current build log.

Finally the other significant change I have had is external data is no longer directly embedded within the log. This was something that worked when we played around with fixing the out-of-memory issue (unfortunately it never got implemented) so we will include it in vNext. Instead of directly embedding the data within the file we add a reference to the file instead.

The advantages of this approach is our build logs will be a lot smaller as it now contains the build summary instead of all the details. It also gets rid of the problems around XML encoding –currently if the source file was invalid XML (or not XML at all) it would be put in a CDATA section. And since the files are linked to a task it is easy to see which task generated the file – especially if there are duplicates of the task type within the project.

The main disadvantage to this is the client will need to first parse the build log to get the names of the relevant files and then get the relevant files from the server. This could be troublesome when generating the build summary but given that the build summary is static we could look at some way of either caching this on the clients or pre-generating the summary on the server.

So this is a look at some of the differences between the current and new build logs, in my next post I’ll look at some of the nuts and bolts of how the log is generated.

CruiseControl.NET – vNext – Invokable Methods

Now that we have a naming standard for locating any item within the configuration and a standard API for querying and invoking the available actions the next question is how do we expose new actions? As stated earlier I want this to be as easy as possible for people to add new actions (unlike the current state that requires modifying multiple locations!)

To achieve this there are two parts that need to be done.

First to pass arguments and return values a sub-class of BaseMessage is needed. There will be a few default sub-classes that will hopefully provide most of the required functionality but it will also be possible to add new sub-classes as needed. These classes will allow passing around the information between the client and the server plus a few extra bits and pieces of plumbing (e.g. security information.)

The second part is to implement the method and add a RemoteActionAttribute to its definition. This method needs to take in a sub-class of BaseMessage as the argument and return a sub-class (it can be the same but it can also be different as required.)

And that’s it – hopefully nice and simple

There is one additional step – add an optional DescriptionAttribute. This will be used by the framework as a human-readable description that gets passed around.

For example for a ForceBuild action we could have the following definition:

[RemoteAction]
[Description("Trigger a build remotely.")]
public BuildMessage ForceBuild(ProjectMessage request)
{
    // Implementation omitted
}

Actions can be exposed on any item within the configuration (e.g. server, queue, project, tasks, triggers, etc.) The invoker will use the universal name to find the correct object and then either query or invoke the actions. The List method will check for all methods that have the RemoteActionAttribute and return the name and description (if present.) The Query method will add the input and output message types. And finally the Invoke method actually calls the method passing in the arguments and returning the result. Under the hood this uses Reflection to do all the dirty work – but that is the subject of another post.

So up to this posts I’ve covered the basics of the communications. The next question (and my next post) is how does this get exposed to the client? I’m looking at using WCF for the default implementation but it will be generic enough to cover additional protocols as required. So stay tuned and in my next post I’ll cover the implementation of a communications channel on the client.

CruiseControl.NET – vNext – Three External Methods

Another area that is currently challenging to work with is exposing new functionality to external clients. On the server-side along there are three classes and an interface that need to be modified alone just to expose the functionality, let alone actually adding the functionality or modifying the clients. In addition modifying the interface causes other parts of the code to break (mainly in the unit tests) so it is something that I usually try to avoid.

To simplify things in vNext I’d like to only expose three methods – these methods would then expose any additional functionality we want without having to modify the actual communications infrastructure. We already have a version of this with the messaging infrastructure that was added in 1.5 but I’d like to take this a step further and also handle some scenarios that the messaging does not handle.

So in a nutshell the three methods I am proposing are:

• List
• Query
• Invoke

These three methods provides the basics of a communications framework where any item in a server can expose functionality. I’ll come back to this in a minute but first let’s go over these three methods.

The List method will list the possible actions that are possible for an item. The method has only one parameter – a universal name of an item. The communications framework will look up the item based on its name, retrieve its type and then get the possible actions that are available for the current user. For example a project type item might allow GetStatus, ForceBuild, AbortBuild, etc. actions while a queue type item might have GetPendingRequests, CancelRequest, etc. This method will return a list of actions – each action will have a name and a description.

The Query method will provide the details on an action. Like the List method it requires a universal name and it also requires the name of an action. If the action is available and allowed it will return the description and the input and output parameters. If the action is either unavailable or not allowed for the current user it will return nothing.

Finally the Invoke method is what is used to call an action. It requires the universal name of an item, the name of an action and its required input parameters. It will invoke the action and then return whatever output parameters are generated. If the action is not available or not allowed then an error will be generated and returned.

These three methods provide a way for external clients to not only run actions on the server but to also determine whether an action is available. For example force build in 1.5 or later versions can be locked down with security but the dashboard has no way of knowing this. The only way a user can find out if they are allowed to force build a project is by clicking on ForceBuild which will generate an error in return – not very nice!

With the List and Query methods the client can check to see if a user has access to the action, maybe as part of the UI generation or as a sanity check before invoking the action. The description can also be used to return help information to the user as required. It will also be possible in future to generate dynamic interfaces where the list of available actions is dynamically generated based on what the server returns, the user selects the action and is prompted for the input arguments and then performs the invoke.

The other part to having these three methods is any item on the server can expose methods – not like the current state where it is primarily projects that have methods with a few additional methods floating around. Using the universal names we can list the possible actions on any item and invoke them. For example there are some items within a project that store state: e.g. the Subversion source control block can store the last revision number. We could add an action to this block to retrieve this state and return it to the user and we could add another action to allow the user to update the state. By having it on the level of the item only the item needs to know about the actions (we don’t need to keep bloating Project with actions for child items.)

Now I have covered two pieces of the communications framework I’m wanting to implement – universal names and the public methods. The third part is how to tie these together – how can we expose actions on various items and have them picked up by the communication structure. As you can imagine this is the heart of the new structure so I will cover it in my next couple of posts.

Stay tuned…

CruiseControl.NET – vNext – Universal Naming

Another area that is challenging in the current version of CruiseControl.NET is communications. To add a new message that will be used in communicating with a server requires modifications in multiple places – both on the server and the client (although some of the changes made in the past couple of years has made this easier on the client side.)

In addition CruiseControl.NET only supports one protocol for talking to the server – .NET Remoting. There is a web-based protocol but this requires an installation of the dashboard which acts as an intermediate. And as the bug reports show this has caused some other problems. There is an WCF extension that can be used but because extensions were tacked on later not many people know about them – especially as they are configured via the app.config instead of ccnet.config!

For vNext I’d like to take this another step further and abstract away more of the communications infrastructure. This will allow for additional communications protocols natively on the server. Plus we can handle from the start some of the issues that have been raised in the current web-protocol.

These changes will involve a number of different components which I’m planning on covering in the next few posts. In this post I’m going to start on a simple that allows a lot of functionality: universal naming.

Currently we have a bit of a mixed bag with the current versions – we can access some items remotely; others are more hidden. For example it is very easy to access and work with projects – the ever present project name servers as a very useful identifier. It is possible to get some information about queues and tasks but normally this is accessed via a higher level item (e.g. the server snapshot or via a project.) If I wanted to find out how an MSBuild task on a project is configured I would have to get the whole project’s configuration and then extract just the task I wanted – not very obvious.

What I would like to propose is a universal naming standard that allows almost any item within the server to be addressed. This will make it different from how things currently work but it will allow much more control over what is accessed.

The base of this naming is to use non-standard URNs. These will be non-standard in they will not use an officially defined namespace id but otherwise will work just like a standard URN. The namespace will be ccnet – so all identifiers will begin “urn:ccnet:”. As per RFC 2141 “urn” is case-insensitive and “ccnet” will be also.

The generic form of a ccnet urn is:

urn:ccnet:server:path-to-item:item

server and item will always be required. server is either the unique name of a server or its address. For example for the server ccnetlive located at http://ccnetlive.thoughtworks.com server would be either http://ccnetlive.thoughtworks.com or ccnetlive. If ccnetlive is used then the mapping to the URL will need to be stored somewhere else (more on this in a later post.)

item is the unique name of an item. Which item depends on the path that has been entered. To help with resolving an item identifier I think we need a few rules around names within the configuration:

1. All item names within a level of configuration must be unique
2. All projects must have unique names that cannot be the same as another root-level item
3. All project items must have either a single instance or a unique name

Rule #1 basically means that at each level the name will identify a unique item. For example a level might be the root level of the server configuration or it might be the child items within a queue, tasks within a project, etc.

Rule #2 means we have an easy way to identify requests for projects (more on this in a minute.) Ensuring that project names do not exist at the root (unless the project is at the root) removes any confusion between referring to a project or a root level item.

Finally rule #3 means we can refer to individual items within a project. The single instance part means if an item is not named then there cannot be any other instances of that item type. For example if there is an unnamed MSBuild task then there cannot be any other MSBuild tasks within that project or project item.

Returning back to the URNs if we had a project called ccnet-trunk on ccnetlive the basic URN would be:

urn:ccnet:ccnetlive:ccnet-trunk

The final part of the URN is the most interesting part – a path to an item. Coming back to our earlier example of an MSBuild task within a project we could reference this task by using the project name as the path:

urn:ccnet:ccnetlive:ccnet-trunk:msbuild

In this example the path is “ccnet-trunk”. The power of this comes when we start looking beyond just the simple project/task structures. For example if we wanted to look at a queue within a queue we could have something like:

urn:ccnet:server:outerqueue:innerqueue

Here the item is innerqueue and the path to it is outerqueue. This means only projects need unique names – other structural items can have duplicate names as long as they are not duplicated under an item – we could have another item called innerqueue that exists under an item called roundrobin for example.

We could also use this syntax for looking at items that belong to other items. If we had an MSBuild task within a Sequence we could use:

urn:ccnet:ccnetlive:ccnet-trunk:sequence:msbuild

This syntax would allow us to access any part of a server’s configuration easily and consistently.

There is one additional area that needs addressing and that is for individual builds. For example we might want to retrieve the status of a build of a project. Now this by itself has a couple of issues: first the build is more of a filter rather than an actual part of the address and second there is currently no easy way to address every build. Projects do have labellers that generate unique labels for a build, but these are only guaranteed unique for successful builds (i.e. the labels can be re-used from failed builds.) To get around this the current version uses a combination of a date/time stamp, the build status (passed/failed) and the label for uniquely identifying a build!

So to address the first issue I think we should use a different symbol for identifying the build, and I am proposing “->”, this would be appended to the end of the URN. For the second issue I think we should use either the date/time stamp or the build label for the name of a build.

For example if there was a build for the above project that started at 13:02:14 on 12-Dec-2010 we could use the following URN for the build:

urn:ccnet:ccnetlive:ccnet-trunk->20101212130214

The date/time stamp should only go to the nearest second (otherwise it gets too long) and we should enforce that a project cannot start building more than once a second.

Alternately if the build was successful and generated a final build label of 2.0.1 then we could use:

urn:ccnet:ccnetlive:ccnet-trunk->2.0.1

This would also be usable with tasks and other project items:

urn:ccnet:ccnetlive:ccnet-trunk:msbuild->2.0.1

Using the alternate symbol (->) instead makes it easier for address resolution. If we used colons for both the build and the item name when we would need to do a look-up to decide whether it is a build name or otherwise.

So this post has covered a naming standard we can use for addressing different parts of a server. As a preview there are two additional parts I’m planning on covering for communications:

• The Three External Methods: looking at how we can condense our external surface to use three methods (List, Query and Invoke)
• Invokable Methods: looking at how we can expose any method on any item so it can be invoked by an external client

So if you don’t know why I want to add universal naming to vNext stay tuned and (hopefully) things will become clearer.

And as always, I’m happy for feedback, so let me know what you think.

Unit Testing Logging

As part of the upgrade we’re planning on making to CruiseControl.NET we are using NLog for logging messages. We are also aiming to increase the unit test coverage of the code. So the question arises – how can we test that logging is actually working as it should be?

There’s actually two parts to this question:

1. Are the logs being generated correctly?
2. Are we putting the right stuff in the logs?

Question #1 is really outside our scope – considering the number of different ways the logging can be configured we’d have to duplicate all the tests in NLog itself. So rather than doing this we’re just going to assume that the tests in NLog will pick up any errors with log generation.

That leaves the second question – how can we know we are putting the right stuff into the logs? The short answer is we can’t – but then there is no way that we can guarantee that any code we write is 100% correct!

Instead what I’m interested in doing is checking that we are logging some expected stuff. In other words how can we intercept the logging to ensure that we are logging the expected messages?

To answer this question we need a little bit of background. Logging in NLog can be configured either via a configuration file or via code. Now the recommended way is the config file so it can be changed without forcing a re-compile. But for us the way that we will use is via code.

NLog also has the concept of Targets: where the output is written to. There is a wide variety of different targets for different scenarios (console, file, e-mail, database, etc.) Of interest in unit testing is the Memory target. This will write the log messages to an array in memory.

So all we need is some way to wire up the logging to a memory target and then check the messages that are being written. It would also be nice if the unit tests reset the logging after they’ve finished just in case there has been some configuration in the unit test config file.

To make this easy I have added a helper class to our unit tests that simplifies this whole process. This is a disposable class with a static method for initialising the whole process. To use this class looks something like:

using (var intercept = LogHelper.InterceptLogging(typeof(TypeBeingTested)))
{
    // Tests go here
    var expected = new[]
                        {
                            // Expected messages go here
                            "LogLevel|TypeBeingTested|Message"
                        };
    CollectionAssert.AreEqual(expected, intercept.Messages);
}

Since the class implements IDisposable it can be used in a using statement. This will ensure that the configuration is reset after the tests have finished.

The static method takes in an array of types. These types are the loggers that should be intercepted – after all there can be other messages occurring that we don’t want to intercept.

To actually check the results we need to compare the Messages property on the class with the expected messages. This is just a facade property that exposes the messages from the underlying memory target. As in the example above the messages consist of three parts: loglevel (e.g. debug, info, error, etc.), the loggger (this will be one of the types in the array) and the actual message.

Nice and simple, without adding too much overhead to the testing process.

Let me know if anyone is interested in the actual code behind LogHelper and I will post it. Otherwise it is in the Subversion repository for CruiseControl.NET.

CruiseControl.NET – vNext – Triggers

Triggers in the current versions are fairly straight-forward. All they have to do is implement ITrigger which defines one property and two methods. In vNext triggers will still have these properties and methods but with a couple of extra features. But before I go over the changes let’s review what triggers there are in the current versions.

The current triggers can be roughly divided into two groups – those that “fire” integrations and those that modify integrations. When I say “fire” I roughly mean that the trigger is reasonable for initiating an integration request. The triggers in this category are:

• CronTrigger
• IntervalTrigger
• ProjectTrigger *
• ScheduleTrigger
• UrlTrigger *

An asterisk next to the trigger means it contains an inner trigger. The trigger itself first checks the inner trigger; if this trigger fires then it does any additional checks which may result in it  being fired.

The second category are those triggers that modify the integration request from another trigger. These all contain other triggers:

• FilterTrigger *
• MultipleTrigger
• ParameterTrigger *
• RollUpTrigger *

The one unusual trigger is MultipleTrigger. This actually contains multiple other triggers (it is the only one that can contain multiple triggers.) Technically it does not really modify an integration but it does contain some logic around whether the integration requires just one or all of the triggers to fire.

The two methods that need to be implemented are Fire() and IntegrationCompleted(). Fire() is called when the trigger needs to be checked. If the trigger has been tripped then this should return an IntegrationRequest; otherwise it should return null. IntegrationCompleted() is called after an integration has been completed – this allows the trigger to reset itself. There is one thing to note about this arrangement – although IntegrationCompleted() allows the trigger to reset this is only for resetting internal state. A new IntegrationRequest is actually generated every time Fire() is called (assuming the trigger has been tripped.)

The final part of the current triggers is the NextBuild property. This returns a DateTime specifying when the next build will be either checked or triggered. It is not documented anywhere (at least that I can see) but if the trigger does not have a next build date/time it should return DateTime.MaxValue.

So how will triggers change in vNext?

First they will have the standard Validate() method. This will be called when the Project itself is being validated (i.e. after it is loaded but before it is started.)

Second they will have Initialise() and CleanUp() methods. The Initialise() method will be called when the project starts – this will allow the trigger to reload any previous state or initialise itself to an initial state. CleanUp() is called when the Project has stopped – allowing the trigger to perform any clean up work.

The Fire() method has been renamed to Check() and it has become a sealed method. Internally this calls an abstract method called OnCheck() which should perform the actual check. The reason for this change is the trigger will store the request once it has been tripped – subsequent calls to Check() will return the same request until the trigger has been reset.

IntegrationCompleted() has been renamed to Reset(). This will clear the previous request and then call OnReset() to allow any custom state resetting.

These changes mean that triggers now perform more like a ManualResetEvent – once tripped the trigger will remain tripped until reset. This should resolve some of the issues that have been identified with how MultipleTrigger works with some of the other triggers (e.g. ProjectTrigger and UrlTrigger.)

The final change is NextBuild has been renamed NextTime and changed to a nullable DateTime. If there is no next time then this should return null.

Otherwise triggers will continue to be used in the same way – they will be called on a regular basis (~2Hz.) When a trigger has been tripped it will start an integration – it is up to the tasks to define what happens after a trigger has been tripped.

CruiseControl.NET – vNext – Source Control Blocks

Source control blocks will be roughly the same in vNext. The main difference is they are external to the build process, to use them one or more source control tasks will be needed.

Internally they will be roughly the same. Like tasks they will inherit from a new base class – SourceControlBlock. This class has one standard property (Name) and the following methods that can be overridden:

• Validate()
• Initialise()
• GetModifications()
• Label()
• GetSource()
• CleanUp()

Validate(), Initialise() and CleanUp() are the same as the task-based versions – they will act the same way and be called at the same time. The main difference is these are directly overridable (in the task the OnMethod() is overridden instead.)

The other three methods need to be implemented by a source control block as they provide the actual functionality. Each of these methods takes in a MethodParameters instance (e.g. GetSourceParameters, etc.) This allows us to easily add new parameters later on without breaking other implementations (something we currently have a problem with.)

To use the source control blocks there is a base class called SourceControlTask. This contains the Use property and a helper method to get the source control block (and in time it may have other helper methods.) Using this makes implementing the source control tasks very easy. For example the current implementation of GetSource is:

protected override IEnumerable<Task> OnRun(TaskExecutionContext context)
{    
	var block = this.GetSourceControlBlock();    
	var parameters = new GetSourceParameters();    
	block.GetSource(parameters);    
	return null;
}

Note: this needs to be expanded to correctly pass the parameters – I haven’t gotten that far yet.

Other than these changes source control blocks will continue to work in the same way as they do currently. They can have their own parameters that are configurable and their different implementations depending on the actual source control provider.

Expanding on Validation from “Task States”

Another question I had was around validation – how will this work in vNext. The asker had obviously looked at the code I’ve been working on as currently it doesn’t have much in the way of validation functionality (a “do or die” approach to use his words.)

This is in contrast to how validation currently works within 1.5/1.6. These versions have a helper interface that allows both warnings and errors to be handled. A warning will merely be displayed in the log – it doesn’t stop the server – while an error will prevent the server from running. Additionally the validator handles both errors and warnings and displays them for the relevant items.

The good news is I plan to continue this in vNext, especially as the validator is a very useful tool. To handle this I’ve added a new interface to the code called IValidationLog. This works in a similar way to the current versions. I’ve also changed the signature of the Validate() method to use an instance of this interface:

void Validate(IValidationLog validationLog)

Like the interface in the current versions that this interface is based on this interface has a method for adding errors and another for adding warnings. As an example of how this works here is the implementation of Validate() in ServerItem:

public virtual void Validate(IValidationLog validationLog)
{
    // Everything must have a name
    if (string.IsNullOrEmpty(this.Name))
    {
        validationLog.AddError("The {0} has no name specified.", this.ItemType);
    }
}

As name is a required property for any server items this will add an error.

Hopefully this clears up how validation will work a bit.

Expanding on “Running a Task”

Ok, I’m back from holiday and slowly catching up on things (I had pre-prepared a series of posts hence why it appears that I haven’t been away.) Since my e-mail reader displays the latest e-mails first I saw a comment by Ruben asking why we should change to returning an IEnumerable of tasks in OnRun(). In his words:

I’m just wondering what the benefit of returning an array(enumerable) is when calling OnRun. It does not seem/sound right. When a function is named OnRun, I would expect no return value, making it void, or maybe returning a result. So returning ‘other’ tasks feels unnatural.

Since it doesn’t sound right it means that I haven’t explained the why clearly enough

Basically the why can be summed up in one word: “Control”!

But before I go more into the why, first let’s cover a bit of background. In CruiseControl.NET there are two general categories of tasks – those that execute and return and those that execute other tasks (containers.) The majority of tasks are execute and return type tasks (e.g. MSBuild, NAnt, etc.) They do their work and that’s it – nice and simple. However from 1.5 onwards there is a new category: those that execute other tasks. These include the parallel, sequential and conditional tasks (plus a few others.) The change I am proposing is primarily for this second category.

In the current versions of CruiseControl.NET (1.5 and 1.6) the parent of the task is responsible for executing any children. If the parent is the project then the project executes the tasks; if the parent is one of the container tasks then it is responsible for executing the task. For these container tasks the project (or any other parents) has no idea of what is happening. As far as it is concerned the task is just a standard execute and return task.

Now this can cause a problem: what if the project wants to do something else while the tasks of a container are running? At the moment the only option it has is to wait until all the tasks within the container have finished (actually there is another option – abort the task – but it is not very nice!) One scenario that I want to implement where this is a problem is pausing a build (more on this below), another scenario is looking at distributing tasks across multiple servers, another is allowing different actions for a task if it has failed, etc.

Ok, now back to the why. For the execute and return tasks this change has absolutely no impact (well, maybe one, it forces them to add a return null to the end of the method.) So if all we had was this category I wouldn’t suggest this change. But for the second category this change makes a huge difference! Having the container task return the child tasks to the project to execute them means the project knows exactly what is happening. If the project needs to be paused it can just wait until whichever task is currently running has finished and then not start the next one until the project is resumed. If the project is distributing tasks to other servers again it knows which task is currently running and can distribute the child tasks to different servers if desired. In addition the code for executing a task is centralised – no need to duplicate the code across different tasks (like we currently have.) So adding the failure options (and the conditions) is very easy – it only needs to be added to one place.

So hopefully this helps explain why I want to make the change. As stated above it makes no difference to the execute and return tasks – it is primarily for benefit of the container tasks.

The second question Ruben had was:

Also what is the benefit of ‘pausing’ a task?

He then mentioned some pain areas we could have with people manually pausing a project (we won’t really pause tasks – the execute and return tasks will not be effected.) And on the whole I tend to agree with his comments – if a project is paused then people are likely to forget to resume it and this will cause problems.

However I’ll admit – I wasn’t thinking of allowing people to manually pause projects (although some people might actually want this.) My idea for pausing projects comes from two areas:

1. Somebody wants another more important project to “interrupt” a current build and then the current build resumes
2. One or more tasks in a project are sent to another server to execute and the build administrator does not want this to prevent other builds

For the first scenario a user would send a priority force build request to the server (assuming they have sufficient permissions). The project would receive this request and ask to start integrating. If the request is not immediately granted then this means another project is building (and they are both in the same queue or similar.) The forced project would then send a pause to the building project and re-ask to start integrating. When the current building project pauses the forced project would integrate and when finished it would tell the original project to resume. So in this scenario the pause and resume would both be automatic without the user having to do anything (or forget anything.)

The second scenario is a bit more interesting. Again there would be two or more projects in the same queue (or similar structure.) If the first project has remote tasks it would send off these tasks and pause itself. This would allow the second project to integrate at any time. When a remote task on the first project is returned it would send a pause request to the second project and wait. Again the second project would pause and the other project can continue. If the first project finishes then it would resume the second project. Or if another a remote task is encountered it would again pause allowing the second project to continue.

Now I should mention at the moment these are merely some ideas I have around pausing a project – at this point I have no idea on how to implement them (especially the communications between projects) – but at least if we know about them we can plan on implementing them.

Finally as I was writing this I thought of a third scenario where pausing might be useful – a task might fail because a resource (typically remote) is not available. Rather than failing the entire build the project might just pause for a specified period and then retry the task.

So hopefully this clears up some of my intentions a bit. The overall goal of these changes is to make the codebase simpler for future work so if this sounds too complicated we can drop it

And I’ll continue working my way through my e-mail backlog and answer any other questions, stay tuned…