Parallel and Sequential Tasks

That’s an Old One

Now that 1.4 is officially finished, it’s time to start gearing up for 1.5. Already we have a large range of improvements (security, dynamic parameters, communications improvements, etc.) However, while these improvements are nice, we also need to review any outstanding issues to see what we can resolve.

One of my pet dislikes is patches that get ignored! People have put an effort into enhancing CruiseControl.NET and then nothing happens. Personally I think this is very sad, although now that I have dev rights to Subversion I can understand how this happens. That aside, I decided to spend some time reviewing the old issues to see what patches can be applied.

Back in August of 2007 Patrick Boyd submitted a patch to allow tasks to run in parallel (CCNET-964). This task is fairly straight-forward – it has an array of child tasks to be run. When the task is run, it starts up a new thread and fires off each task. So, I thought I’d go and commit it (especially as there are five votes for it).

Some Revisions

While the original patch “works”, there are some issues it does not take account of. The one that got me is the way it handles the results from a child task – it just hands the parent result to each child. Unfortunately, this can cause issues as the result is not guaranteed to be thread safe.

So I when through and revised the patch and (hopefully) improved it. So, here is what I have changed:

1. Changed to use ThreadPool for generating the threads. I changed to this to help manage the number of threads used by CruiseControl.NET. Already we fire up one thread per project (nasty!) and now we’re firing up more threads for parallel tasks. Using the ThreadPool helps ensure we don’t overload the system (too much!)
2. Generated an instance of IIntegrationResult for each task. This is actually a clone of the parent result, when all the tasks have finished running, the main task goes through and merges all the results (this is actually implemented in IntegrationResult). This guarantees the thread safety of the results.
3. Added some error handling. When a child task now runs, I wrap it in a try/catch block. I did this because I got some weird results in the unit tests – mainly because the ManualResetEvent instance for the thread was not being set! Adding the try/catch block helps with this.
4. Generated lots of log entries. Well, not that many, but enough to hopefully show what is happening within this task. These log entries now provide a view of what is actually happening inside the task (and within each child task thread). I could probably add some more logging, but I tried not to go overboard.

Additionally, I fully unit tested this code. This was a combination of TDD and TAD – I added some extra logic after I wrote the tests that introduced some more scenarios. So I went back and added tests for more.

What About Sequential Tasks

In his original patch Patrick provided a parallel and a sequential task. Originally I didn’t want to add the sequential task as I didn’t see the value of it. However, I eventually saw why he added it – within the parallel task you might want a child task that has several tasks. But they should all run in sequence.

So I also added a sequential task. This uses a lot of the same code as the parallel task. Actually, there are two main differences between the two:

1. Parallel uses threads to run each task, sequential uses the project thread.
2. Sequential has the ability to cancel pending tasks if a task fails, parallel just runs everything.

Both tasks use a clone of the results, both have improved error handling and both use logging extensively.

An Example

As an example of how these work, here is a small block that shows how parallel and sequential tasks work together:

  1: <tasks>

  2:   <parallel>

  3:     <tasks>

  4:       <sequential continueOnFailure="false">

  5:         <tasks>

  6:           <exec>

  7:             <executable>RunTest.cmd</executable>

  8:             <baseDirectory>D:\Temp</baseDirectory>

  9:           </exec>

 10:           <exec>

 11:             <executable>RunTest.cmd</executable>

 12:             <baseDirectory>D:\Temp</baseDirectory>

 13:           </exec>

 14:         </tasks>

 15:       </sequential>

 16:       <exec>

 17:         <executable>RunTest.cmd</executable>

 18:         <baseDirectory>D:\Temp</baseDirectory>

 19:       </exec>

 20:     </tasks>

 21:   </parallel>

 22: </tasks>

Of course, there is nothing stopping people using these tasks within other tasks.

In Conclusion

So, sorry Patrick for taking so long to get to your patch. Hopefully this now provides an improved version of the original patch.

And hopefully this will be useful to other people as well (I can already think of some scenarios at my work where it will be useful!)