Developer Documentation

Have you ever wanted to write plug-ins or extensions for CruiseControl.NET? Do you feel like it is an uphill battle to get started? Unfortunately, like a lot of open source projects, the documentation in CruiseControl.NET is not always the best (something to do with the developers’ mentality!) One of the areas in CruiseControl.NET that is very, very poorly documented is how to write extensions for CruiseControl.NET.

The good news is I have found a couple of volunteers who are developing extensions, and are happy to proof read any documentation I put together. So, in-between trying to resolve almost impossible issues in CruiseControl.NET, I will be writing up on how to develop extensions for CruiseControl.NET.

At the moment, here is my proposed outline for what I am writing:

• Introduction
• Overview
• Basics
• Tasks/publishers
• Triggers
• Source control blocks
• Labellers
• State managers
• Server extensions
• Web dashboard templates
• Web dashboard plug-ins
• Common components
• Tips & tricks
• Further information

This will probably change as I am writing it, but hopefully it covers most of the areas of customisation in CruiseControl.NET (with the exception of security – but that’s an entirely different challenge!)

If you are interested in participating in this (either contributing, proof-reading, etc.) let me know and I will get in touch.

CruiseControl.NET Converted to .NET 3.5

Over the week-end we finally upgraded CruiseControl.NET to use .NET 3.5. At the same time we branched the trunk and updated the version number for the trunk to 1.6.

This is good news in two ways:

1. This means that the 1.5 release is finally marked as finished – there will be no new functionality added to it. There is still the potential for bug fixes, but hopefully these will tail off and we will be able to finally close off the 1.5 release (like we have the 1.4.4 release).
2. CruiseControl.NET can finally start to use some of the goodies in .NET 3.5. This includes Linq, some multi-threading improvements, ASP.NET 2.0, etc.

At the moment CruiseControl.NET does not use anything from .NET 3.5, but we can look forward to this changing in the near future

When we were debating whether to change to .NET 3.5 or not, we had one person ask why not wait a few weeks for the .NET 4 release and go straight to that? The answer is very simple – we are trying to ensure that CruiseControl.NET also runs on Mono. This means people on non-Windows OS’s can use it as well. Unfortunately, from what I have seen, there is no timeframe for implementing the new .NET 4 functionality into Mono yet, so we will have to wait and see what happens on that front.

But when Mono does have .NET functionality, I will be pushing to upgrade faster this time around…

My Vision for CruiseControl.NET

Introduction

I’ve been a bit quiet on the CruiseControl.NET front lately – mainly due to a change in job, plus I’ve taken on post-graduate studies. Unfortunately, this has also meant that work on CruiseControl.NET has pretty much come to a halt as well The other core developers obviously have the same lack of time.

However, while I haven’t had much time for coding, I have had some time for thinking. So I thought I’d note down some thoughts about where I’d like to see CruiseControl.NET go in the future.

Note: at this point I want to stop and make a disclaimer – I am not the project lead, lead developer or anything of the sort for CruiseControl.NET, I am just a contributor (although yes, a fairly active one). As such, this post is purely my thoughts on future direction – there is no guarantee that any of them will be done.

1.5: Current Release

We are currently in RC1 for 1.5, but not much is happening. I’d like to draw a line in the sand – do what we can within a set timeframe and then have that as the official 1.5 release. I know it will still contain some bugs and unresolved issues, but I do not think they will be resolved anytime soon (unless one of the other developers suddenly gets some spare time!)

For my part, I’m hoping to spend some time dealing with the critical concurrency issue we have – this is basically where two external tasks are conflicting and causing one of them to halt until the other is finished. However, given the difficulty of this, I can’t promise any resolution.

1.6: Next Release

Again I’d like to draw a line in the sand and propose a cut-off date (at the moment I am thinking beginning of next year.) This release will include a number of the patches we have received but were unable to include in 1.5, plus some fixes on out-standing issues.

It will also be the first .NET 3.5 release of CruiseControl.NET – I’ve managed to get enough of a consensus within the core developers to make this happen. Currently we are resolving some issues and then we will switch to a 1.6 development mode.

The other major item for 1.6 is the inclusion of the CCNet Conditional Plug-in. This is an awesome extension that has been developed by Lasse Sjørup. Basically it allows conditions within CruiseControl.NET (task sections only). It is extensible and already covers a wide range of options. This plug-in is currently available at http://ccnetconditional.codeplex.com/, but from the 1.6 release it will be included natively!

One of the other developers was also talking about ASP.NET MVC 2.0 – so perhaps there is a possibility of finally replacing the current dashboard (and all its home-grown components) with something that is easier to maintain. Just a rumour, but it would definitely be something that I would get involved with!!!

2.0: 2012 (perhaps?)

The next major iteration of CruiseControl.NET will be sometime in the future. There are two major changes I would like to see, both of which are breaking and pretty major.

First, I would like to get rid of the massive in-memory strings that we currently have. This is basically the work that I have done around converting to stream-based logging. Unfortunately this is not in a usable state at the moment, there are still some significant issues to resolve. However this would go a long way to reducing CruiseControl.NET’s memory consumption and it will also speed things up.

Second, I think we should change from NetReflector to XAML for the configuration. This would eliminate another component that we are the sole users of, and thus is not progressing at all. Unfortunately this also has a dependency on .NET 4.0, so we cannot progress down this path until Mono supports .NET 4.0.

The Rest (2.1 onwards)

Some of the other areas in CruiseControl.NET that I think we can work on once we have moved to .NET 4.0 are:

• Removing the dependency on .NET Remoting (perhaps even offering multiple communications channels)
• Alternate scheduling engine (current the only option is queue-based)
• Using the parallel extensions in .NET 4.0 to simplify some of our threading (most of the current code was written pre-.NET 2.0)
• Use MEF to allow additional extension points within the server
• Allow alternate configuration stores (e.g. databases, common configuration server, etc.)
• Distributed builds (peer-to-peer?)

Anyway, these are just ideas at this time, but I think it is good to have a vision for the future

So this is what I would like to see happen in CruiseControl.NET. If anyone else has anything they would like to contribute, or even if they would like to get involved with the project, let me know and I will see what I can do.

Fun with Concurrency

As part of my studies I am researching multi-processor (or concurrent) programming. One potential issue in concurrent programming is two threads (or processors) attempting to access the same variable at the same time. When both threads are reading, this is not an issue, but if one (or both) are writing this can lead to corrupted data being retrieved.

For example if one thread wrote the value 7 to a variable. Later, at the same time another thread write the value –3 to the same variable at the same time as the original thread is writing. The question is what would the thread get? We would hope it would get either 7 or –3, but it is equally likely it could get something like –7, a corruption combining the two values!

After reading this, I wondered if this was possible in CruiseControl.NET. And the answer is yes – it is very easy to get something like this!

Each project runs on its own thread, but as long as it only updates its own data it is ok. But, any requests for information come in via .NET Remoting (e.g. if the dashboard wants to update its view or CCTray is polling the project status). When the request is received CruiseControl.NET handles it on a separate thread from the project thread. If the project is also updating at the same time the read is happening, there would be data corruption!

To test this, I modified the code slightly. I added some extra tracing statements – basically to show the start and end of GetProjectStatus() and every call to LastIntegration. I also modified the integration status so it has a unique ID that changes every time the status is updated (this is a GUID). I then ran some tests to see if this ID changed while GetProjectStatus() was running.

What I was looking for is very simple, if the ID changed, then GetProjectStatus() will be returning corrupted data. After a number of attempts (plus adding a small Sleep into LastIntegration) I was able to get the following:

[20:DEBUG] Start => GetProjectStatus()
[20:DEBUG] Get ===> LastIntegration: a9ba4e605ac1476099e447eaf1175bb1
[20:DEBUG] Get ===> LastIntegration: 840b18e5a5244ebfbbe4066557b0c3a6
[20:DEBUG] Get ===> LastIntegration: 840b18e5a5244ebfbbe4066557b0c3a6
[20:DEBUG] Get ===> LastIntegration: 840b18e5a5244ebfbbe4066557b0c3a6
[20:DEBUG] End ===> GetProjectStatus()

The first Get has a different ID from all the rest – which means it is a different LastIntegration. This shows that the status can have corrupted data (i.e. data from multiple sources!) Now I had to cheat a little to get this, but that fact that it is possible means it can happen! Hopefully though it is very unlikely to happen!

Going forward, what does this mean?

At the moment, I’m not sure. It does raise the possibility of some really subtle bugs in the communications, but whether this is significant, I don’t know. Since no one has mentioned it, it probably doesn’t happen much, but we should probably look into it sometime.

And of course the other question is: what other areas are potentially affected like this?

The joys of concurrent programming…

A XAML Hiccup

One of the neat things about XAML is the ability to use attached properties. These are properties that do not belong to an object, but instead to its parent object. The classic example used is the dock location for a control within a dock panel – this is associated with the control, but actually belongs to the parent.

It turns out that this “feature” is WPF specific (although it also works for Silverlight, but the two are related.) Now this may not seem like an issue, but it causes an hiccup in the XAML serialisation.

To illustrate, here is a XAML fragment that I have been working with:

<Queue Name="TestQueue">
    <Project Name="QueueProject"
         Queue.Priority="0">
        <Project.ExternalLinks>
            <ExternalLink Name="Somewhere"
                      Url="http://somewhere" />
        </Project.ExternalLinks>
    </Project>
</Queue>

This fragment contains an attached property on the project within the queue – Queue.Priority. XamlServices.Load() handles this without any problems – this property can be retrieved and it has the correct value.

Unfortunately XmlServices.Save() completely ignores it!! Here is what the XAML looks like after serialisation:

<Queue Name="TestQueue">
  <Project Name="QueueProject">
    <Project.ExternalLinks>
      <ExternalLink Name="Somewhere"
                    Url="http://somewhere" />
    </Project.ExternalLinks>
  </Project>
</Queue>

Now, all is not lost, it is still possible to serialise the attached property – we just have to use System.Windows.Markup.XamlWriter.Save() instead. That gives the following XAML:

<Queue Name="TestQueue">
  <Queue.Elements>
    <Project Name="QueueProject"
             Queue.Priority="0">
      <Project.ExternalLinks>
        <ExternalLink Name="Somewhere"
                      Url="http://somewhere" />
      </Project.ExternalLinks>
    </Project>
  </Queue.Elements>
</Queue>

But now we have another problem – the content properties have re-appeared!

So now I’m stuck – XamlServices.Save() misses out important data, while XamlWriter.Save() adds unnecessary elements (I haven’t seen if it has any of the other bloatware elements yet). In this scenario I have to go with XamlWriter.Save() as the attached properties are critical!

Perhaps this is just an issue with the RC and it will be fixed in the release – after all XamlServices.Load() handles the attached properties. I certainly hope so…

SubMain’s CodeIt.Right and CruiseControl.NET

A New Tool

I’m always on the look out for developer tools that can help my development. And if the tools can be integrated with CruiseControl.NET – even better!

Recently I found another static code analysis tool – but one with a difference. The tool is called CodeIt.Right by SubMain (http://submain.com/products/codeit.right.aspx). This performs a static analysis for C# and VB.NET code. The difference is it also offers automatic refactoring – it can fix the problems that it finds!

I got in touch with SubMain and they were happy for me to integrate it with CruiseControl.NET. So, I have put together an integration package to include it.

The Task

As well as integration with the IDE, CodeIt.Right comes with a command line version. This tool goes through and analyses the code and produces an XML report out the other side. So it was a simple matter to add a new task that calls the command-line version.

Rather than stopping there, I’ve tried to add a bit of extra value. First, the task will automatically merge the XML results file. Since this is integrated into the task itself, it will only merge the file if the task is run. I mention this because it is often an issue with the merge task – if it is put in the publishers section it will always merge the file, whether or not the task that is supposed to generate it ran or not.

Second, After the analysis has finished, the task can be configured to scan the results file and fail if any violations over a severity level have occurred.

To call the task is as simple as:

<codeItRight>
  <project>remote.csproj</project>
  <executable>Tools\SubMain\CodeIt.Right\SubMain.CodeItRight.Cmd</executable>
</codeItRight>

Where <project> is the project to analyse (you can also use <solution> for scanning an entire solution) and <executable> is the location of the command line executable. A more complete example is:

<codeItRight>
  <solution>ccnet.sln</solution>
  <executable>Tools\CodeIt.Right\SubMain.CodeItRight.Cmd</executable>
  <crData>ccnet.crdata</crData>
  <failureThreshold>Error</failureThreshold>
  <profile>FullScan</profile>
  <reportingThreshold>Warning</reportingThreshold>
  <timeout>600</timeout>
</codeItRight>

Dashboard Integration

The second part of this integration is to add a dashboard plug-in. Since the report is in XML, it is a simple matter to convert it to a report.

CodeIt.Right already comes with an XSL-T template to generate a report, but since it is part of the product, I wasn’t able to include it in CruiseControl.NET. Instead I have produced a different report. Initially I tried to duplicate the CodeIt.Right report, but my XSL-T skills weren’t up to the challenge So, instead I have produced a CruiseControl.NET specific report.

This report is similar to the main project grid – it displays all the results in a sortable grid:

Additionally, the results can be filtered, thus reducing the “noise” in the report.

I also added a summary report, which is included on the main build report:

Finally, to round everything off I built a dashboard deployment package, so adding the new reports is as simple as going to the admin console and installing it.

In Conclusion

This was a fairly easy tool to integrate into CruiseControl.NET. The task simply runs the tool and merges the result, with the added benefit of checking the results to fail the build if necessary. The reports were just some simple XSL-T templates, plus a package to make it easy to install!