A Son is Born

This morning at 6:25am, my son, Ethan Philip Sutherland, was finally born. He is a nice healthy young boy, although certainly on the heavy side (especially considering the size of my wife!)

Normally, I aim not to post personal events in this blog, but this event will have a significant impact on my life and the various tasks I do, hence the reason for the post.

I am taking the next four weeks off work to help my wife adapt from a two-person to a three-person family (an yeah, I need the time too!) As such, I probably won’t have much time to work on CruiseControl.Net either, although this will be even more impacted.

I had hoped to have the security branch merged and tested by this time, but due to continuing delays in completing the 1.4.x releases, this hasn’t happened The code is 95% complete for the following features:

• Security
• Dynamic build parameters
• Messaging
• Communications client
• Server extension points
• Build throttling

I say 95% completed, because there has not been much feedback or testing on these new features. As such, I know there will be additional issues or items that I have not thought of. So, these items are on hold indefinitely, as I have no idea of when I will have time to do them. Perhaps one of the other developers will take these features and complete them…

Additionally, there are a few other items I’ve been working on which haven’t been checked into Subversion anywhere. These include the following features:

• Simplified web administration
• Installation packages
• Detailed task statuses for builds
• Package publisher
• NDepend task (plus installation package)
• XMPP publisher
• Dashboard themes
• Extended source control exception handling

These are in various states of completion, depending on where I got these to. Most of these are proofs-of-concepts, so I only developed them to the point where I was happy the concept was valid. The idea was to then commit these into the trunk and finish them.

If I have time, I may add some of these items in, but then it will make the lead developer unhappy as we are only supposed to be bug-fixing.

Finally the VS2008 migration is also on hold. This work has been completed and tested, but thanks to the on-going issues with 1.4.x cannot be merged!

So, you won’t see many posts from me for the next while, hopefully I will have time to work on CruiseControl.Net, but I’m not planning on it for a while.

Thanks for all your support,

A proud (but tired) new dad

Web Administration Simplified, Part II

Previously

This is the second instalment on my web dashboard administration plug-in (read the first instalment here). In my first instalment I started on a plug-in to administer the dashboard.

This plug-in displayed a list of the configured servers, a list of the packages, plus had a few commands to manipulate things. However, the only piece of functionality I implemented was the ability to reload the dashboard (this forces a reload of the dashboard configuration).

In this post, I’ll go a few steps further and add the ability to manipulate servers and packages.

Remote Servers

The dashboard is not the CruiseControl.Net server. It can sit on the same machine, but it doesn’t have to. Additionally, one dashboard can monitor multiple servers – it’s not limited to just one! Therefore, it would be nice to add, modify and delete servers.

The admin view has a button called “Add New Server”. Clicking on this button opens a pop-up within the page to allow the administrator to configure a new server:

This has the same details as the config file. Both of the text boxes (name and URI) are required, saving without entering these will display an error. Clicking on save will add the server to the config (assuming everything is ok), update the list and reload the configuration:

Clicking on any of the servers will display the details for the server and allow them to be changed:

There is also a new button on this pop-up: “Delete”. Clicking on this button will first prompt if you really want to delete the server:

And then delete it (if the administrator clicks on “Delete”):

So, an administrator now has full control over the servers – add, edit and delete (or CRUD if you prefer).

Packages

I’ve also imported my previous package work into this plug-in. Clicking on the “Load Package” button allow the user to select a package:

And then import it:

This will load the package to the server, but it won’t install it. Instead the administrator needs to either click on the “Install” button, or click on the package and click on the “Install” button in there:

Either way, clicking on the “Install” button will install the package to the server:

There is also an option to view the log. This log is the same as the previous log, just in a nice pop-up now:

So an administrator can see what happened

This is what I’ve currently implemented for packages. Looking at the pop-up, there are a couple of extra buttons:

I haven’t implemented the “Uninstall” or “Remove” buttons yet ("Remove depends on uninstall). But this is enough to provide a rough overview of what will happen with packages.

Coming Up

So, I need to do one more post in this set – uninstalling and removing packages. But before I can do the post I need to implement the functionality.

So, stay tuned…

Communications Client and Extensions

A Long Story Short

When I first started development for CruiseControl.Net, I wanted to add a WCF transport mechanism to the product. Now I did achieve this, but it took a fair amount of work. The way I original did it was to provide implementations of the transport classes within CCTray, plus I added a server extension to handle the server-side.

This all involved a lot of work, as I needed to implement all the important methods in the transport classes, plus in the end, this only worked between CCTray and the server – the dashboard was excluded from the process.

Since then, I’ve come a long way, both in my understanding of the code, plus with the modifications I have done. Going back and looking at this, I now realise that there is a much easier way to do this! Interested? Read on…

The Communications Client

Recently I have modified the code of CruiseControl.Net to use a messaging-based structure for the method calls between the client and the server. This involved a modification to ICruiseManager (which I later renamed to ICruiseManager2), and then a whole new set of communications classes. These classes all reside in the Remote project (plus they are also referenced by the Communications project).

When I started to implement the client, I realised that there were lots of methods that I would need to implement, and I’d need to implement the communications for all of them. Rather than doing this, I decided on an alternate approach.

I added a new interface called IMessageProcesser, which contains a method called ProcessMessage(). I then modified the server class to implement this interface. This interface has one purpose – it allows the client to send a message to invoke a method. This means remote clients now only need to implement one method – ProcessMessage().

The communications client is built on this base. The main communications client has all the standard methods on. However, it converts these methods into message objects and then passes them onto a connection instance. This connection instance passes on the message via ProcessMessage(). Now whenever a new method needs to be added, only one class needs to be modified – the communications client class.

To simplify the process of developing new communications channels, I added an interface called IServerConnection. This class handles all the communications between the client and the server. This maps to IMessageProcesser, but allows for both synchronous and asynchronous models (planning ahead for Silverlight).

The main method in this interface is SendMessage(). This converts the message into the however the transport protocol wants it and then send the message. It is also responsible for processing the response message.

As an example, this is how it has been implemented for the HTTP connection:

  1: public Response SendMessage(string action, ServerRequest request)

  2: {

  3:     // Generate the target URI

  4:     Uri targetAddress = new Uri(serverAddress,

  5:         string.Format("/server/{0}/RawXmlMessage.aspx", request.ServerName));

  6:

  7:     // Build the request and send it

  8:     WebClient client = new WebClient();

  9:     NameValueCollection formData = new NameValueCollection();

 10:     formData.Add("action", action);

 11:     formData.Add("message", request.ToString());

 12:     string response = Encoding.UTF8.GetString(client.UploadValues(targetAddress, "POST", formData));

 13:

 14:     // Convert the response into a response object

 15:     Response result = XmlConversionUtil.ProcessResponse(response);

 16:     return result;

 17: }

The initialises a WebClient, adds the form data (the message is serialised to XML using the ToString() method) and then sends it to the server. The response is then processed via a helper method (XmlConversionUtil is a public class that can be used by other protocols) and returned to the caller.

The Server Side

As well as the client, the server often needs to be modified for receiving messages. To handle the actual communications, a server-side extension will need to be added. I have already done this for the original WCF work, but now that things have changed, I need to return and review it.

A server extension must implement ICruiseServerExtension. This class provides a number of methods that the server will call (e.g. Start(), Stop(), Initialise(), etc.), plus it receives an instance of ICruiseServer. This instance contains events that an extension can listen to (e.g. ProjectStarting, ProjectStarted, ForceBuildreceived, IntegrationStarted, etc.)

There is only one issue with using extensions – CruiseServer does not implement IMessageProcesser, so an alternate way is needed to process incoming messages. The good news is this is very easy to do – just instantiate a new CruiseManager instance. CruiseManager does implement IMessageProcesser, and it is a wrapper around CruiseServer.

To show how easy this is, I quickly put together a stub of an in-coming message handler:

  1: using ThoughtWorks.CruiseControl.Remote;

  2:

  3: namespace ThoughtWorks.CruiseControl.Core

  4: {

  5:     class MessageProcessingExtension

  6:         : ICruiseServerExtension

  7:     {

  8:         private IMessageProcessor processer;

  9:

 10:         public void Initialise(ICruiseServer server, ExtensionConfiguration extensionConfig)

 11:         {

 12:             processer = new CruiseManager(server);

 13:             // TODO: Initialise in-coming message server

 14:         }

 15:

 16:         public void Start()

 17:         {

 18:             // TODO: Start in-coming message server

 19:         }

 20:

 21:         public void Stop()

 22:         {

 23:             // TODO: Stop in-coming message server

 24:         }

 25:

 26:         public void Abort()

 27:         {

 28:             // TODO: Abort in-coming message server

 29:         }

 30:

 31:         // TODO: Call this method when an incoming message is received

 32:         private string HandleIncomingMessage(string action, string message)

 33:         {

 34:             return processer.ProcessMessage(action, message);

 35:         }

 36:     }

 37: }

Conclusion

This shows how the new message-based infrastructure makes it simple and easy to add a new protocol to CruiseControl.Net.

Client-side changes are done by implementing IServerConnection, while server-side changes can be implemented via ICruiseServerExtension. There are a number of helper classes that simplify the process, including XmlConversionUtil on the client and CruiseManager on the server.

There is still a bit of work to be completed still, need to modify both CCTray and WebDashboard to use the new communications client. This is mainly grunt work, just taking the time to implement it, so it shouldn’t be too far in the future.

Web Administration Simplified, Part I

The Issue

CruiseControl.Net has been designed around maximum flexibility, to allow an administrator to configure it in an almost unlimited variety of ways. Unfortunately this has come at the expense of ease-of-use. It has been assumed that all administrators are happy to much around in XML configuration files and know what they are doing!

However, this is no longer the case. More and more people are expecting a system that provides an easy to use administration system (e.g don’t make me think!) To add new functionality to CruiseControl.Net can involve a series of steps, often involving copying multiple files and modifying one or more config files. As well as making it harder for people, this also increases the possibility of errors – not good.

Additionally, the dashboard has a little quirk in it – the configuration is cached. If the configuration is manually changed, the web server needs to be restarted. Not exactly the best condition.

A Small Step

A few days ago I wrote a post about adding pre-built packages to CruiseControl.Net (here). In this post I started work on a method to package all the changes into a ZIP file. The all an administer needs to do is import and install the file.

At the end of that post I identified a number of improvements for my prototype. Since then, I’ve done a little more work on this area, plus expanded it to cover more options for configuration.

The Administration Plug-in

The first thing I’ve done is changed the interface slightly. The original plug-in had one purpose, and one purpose only – loading and installing packages.

While this is a good start, it would be nice to administer any of the configuration settings for the dashboard. To achieve this I’ve added an administration view:

This view lists all the server, the packages and provides a few admin options (these may expand over time).

Remote Servers

The servers list shows all the servers that have been configured. Clicking on a server brings up additional information, plus allows the administrator to change the settings:

Clicking on the “Save” button will update this configuration, but I haven’t quite got that far yet:

Changing these settings will be immediate – it will also reload the configuration settings.

Packages

Packages was the concept I started on in the last post. As part of that post I also wanted to add a repository manager. This means that the administrator will work with packages instead of directly modifying the configuration.

With this in mind, I have added a list of packages. The user can install more packages:

They can also click on a package to bring up additional options:

There are a couple of options here: install (not shown), uninstall, and remove.

Installing a package will copy all the files and update the configuration. Uninstall will do the opposite and remove everything.

Remove is the opposite of the load package, but with one small difference – it will uninstall the package if it is installed (load new package will not automatically install the package).

Separating the uninstall and remove functionality allows the administrator to temporarily disable functionality if required.

And Last But Not Least

Some people will still want to modify the config files and do things manually. To make this easier I have added a “Reload dashboard” command. This simplify forces a refresh of the configuration:

This way they can manually modify the config file and then reload it without restarting the web server.

I’m Not Done Yet

This is more of a preview of what I’m doing. What I’ve displayed in the screen shots is working (e.g. the pop-up dialogs, reloading configuration), but not much else.

For the remote servers I need to add the code to modify the server settings. I’m also going to add a delete server command, so people can remove unwanted servers.

I have added the package manager, and refactored some of the packaging code, but it’s not finished yet. I need to re-add the import package functionality (clicking on the “Import” button doesn’t do anything yet), plus I need to wire the “Install”, “Uninstall” and “Remove” buttons. I also need to add the functionality for uninstalling.

And after that? I don’t know. I probably should add some security, but this is different from the other security (i.e. web server-based, not cruise server-based). I also want to go through build some more packages, and of course, I’ll need to test everything.

Stay tuned, more coming soon…

A Duh Moment

I’m spending some time looking at the web dashboard at the moment, trying to make things easier for administrators.

One of the areas that can add a lot of value to the dashboard is some AJAX and dynamic functionality. This means the administrator doesn’t need to send a request, wait for the page to refresh and then continue. Instead a simple request would be sent to the server, the server processes it and send back a JSON response for the client to process.

Now, delving into the code, there is a SiteTemplateActionDecorator that takes a response from an action and wraps it in the site template (all the bits and pieces to get the full page working). Every plug-in exposes a number of plug-ins, which get loaded via CruiseObjectSourceInitializer, and part of the loading involves adding this decorator to each and every action.

This means every action will pass through the decorator and get wrapped in an HTML template. Or does it?

One of the issues with AJAX development is sometimes we only want to return a fragment of HTML, or in my case, a piece of JSON. In this case we don’t want it wrapped in the site template.

Previously when I wanted this functionality, I added a new interface that would cause CruiseObjectSourceInitializer to omit SiteTemplateActionDecorator . While this worked, I always thought it was messy and wanted a cleaner approach. So this time I looked at adding some attributes to do the same thing, but I kept running into issues.

So I kicked around with a couple of other ideas, which is where my duh! moment occurred.

I finally went and looked at SiteTemplateActionDecorator and discovered an interesting piece of information – it only generates the template wrapping when the input response is a HtmlFragmentResponse – anything just gets returns as it was!

So I don’t need to make any changes to the decorators or wrapping functionality, instead I just need to return a non-HtmlFragmentResponse!

To facilitate this I have added JsonFragmentResponse and HtmlAjaxFragmentResponse classes. Both of these will bypass the template wrapping. Now it is possible to get the plug-in to return multiple actions to be processed by CruiseObjectSourceInitializer and not have to worry about the template wrapping.

Now that’s a much cleaner approach than anything I tried

Configuration Woes, Plus Improvements

How Do I…?

One of the common issues with CruiseControl.Net is it can be challenging to configure it – now of it’s parts have any configuration tools (except CCTray). Looking at some of the other CI tools out there (e.g. Cruise, TeamCity, etc.), it is obvious they also have picked up on this. Not only that, but they have done something about it, so much so, that there are people who are changing to these products just because they are easy to configure!

So, this is an issue for CruiseControl.Net, and not something we can easily ignore.

The Bigger Picture

One of the reasons why CruiseControl.Net doesn’t have any configuration tools is it’s flexibility. There is a huge range of possibilities, plus it is pretty easy to write extensions to CruiseControl.Net. So, how do we build a tool (or set of tools) that can cover all the possibilities?

Additionally, there is not just one part of CruiseControl.Net to configure, but multiple. Most people know about ccnet.config, fewer know about dashboard.config and even fewer know about ccnet.exe.config, ccservice.exe.config and web.config. So, the first question is why so many? And can we remove any?

But before I answer this question, what are these files?

First ccnet.config contains the project and queue configurations (and security from 1.5). This is the “main” configuration for CruiseControl.Net, in that it defines the build process. There is no way we can not have this configuration file.

Dashboard.config provides similar functionality, but for the web dashboard. It defines the servers to monitor and the plug-ins to use. Again, it is critical to the system, otherwise the dashboard would be empty! However, there are a few quirks around this file, especially around the way it is cached.

The other three files (ccnet.exe.config, ccservice.exe.config and web.config) are the .NET configuration files. They contain the settings required for things like .NET Remoting, log (i.e. Log4Net settings), a few miscellaneous settings (like where to find the actual config file, etc.) Now, we could remove these files, but it would probably break some things! Therefore we won’t be looking at removing them, although we may try to tidy them up a bit.

Now that I’ve covered the bigger picture, let’s look at improving one (little) area.

Dashboard Plug-ins

The dashboard works on a plug-in model, most of the information that is available comes from plug-ins. If the administrator doesn’t want the users to see some information, they can remove that plug-in. Likewise, new plug-ins can be added to provide additional functionality. Some plug-ins are hard-coded (e.g. the interfaces for CCTray, etc.), but otherwise the administrator has a lot of control of what is included.

To add or remove plug-ins, the administrator needs to modify dashboard.config (in the dashboard folder) and then start IIS. Additionally, if they want to add a new plug-in, they need to copy all the required files (e.g. templates, XSL-T files, etc.) into the right place. This makes adding a new plug-in a fragile process, as there is no guarantee the administrator will get everything right.

Now, in my previous job, there was the concept of a package. A package contained all the necessary files, plus a manifest that described how the files were to be used. Our application would open the package, read the manifest and then correctly deploy the files (at least it would if the manifest was right!) This would be a neat concept for CruiseControl.Net.

This would work in the following way:

1. A developer writes a new plug-in/theme/etc.
2. When it is completed, they zip all the files into a package and then add a manifest saying what needs to be copied and configured.
3. This package is then distributed.
4. An administration retrieves the package.
5. They start up the dashboard, and import the package – this then displays the package details.
6. If the administrator is happy they then install the package, which loads all the files into the correct places and adds the configuration.
7. The administrator is then done

This involves a little more work for the developer, but a lot less work for the administrator! Additionally the risk of user error is reducing significantly.

Sound good? If so, read on, because I have put together a working prototype.

Package Importer

First off, what is the package and what does it contain?

I’ve defined a package as a zip file, that contains all the necessary files, plus an XML manifest. I’ve put together an example package for the download CCTray functionality:

This contains the installer for CCTray, plus the manifest file (there are no templates or binaries required because this is part of the standard install). The manifest file is called “manifest.xml” – this is very important because the package importer needs to find the file, so I’ve based the search on the file name.

The contents of the manifest looks like this:

  1: <package>

  2:   <name>CCTray Download</name>

  3:   <description>Allow a user to download the installer for the CCTray application.</description>

  4:   <type>Plugin</type>

  5:   <folders>

  6:     <folder>

  7:       <location>CCTray</location>

  8:       <files>

  9:         <file>CruiseControl.NET-CCTray-1.4.3-Setup.exe</file>

 10:       </files>

 11:     </folder>

 12:   </folders>

 13:   <configuration>

 14:     <setting>

 15:       <path>/dashboard/plugins/farmPlugins</path>

 16:       <name>cctrayDownloadPlugin</name>

 17:     </setting>

 18:   </configuration>

 19: </package>

This defines all the files (in folder elements), plus the configuration changes to make. I’ve tried to keep the format as simple as possible, but hopefully still cover everything.

The actual package importer is a dashboard plug-in. This means an administrator can remove it if they want to (it’ll also be secured under 1.5 so the user must have admin rights).

The farm level menu for the dashboard will look like this:

There are two things to notice:

1. First there is the new package menu (I’ve highlighted it to ensure people see it, the green bits aren’t in the HTML)
2. There is no CCTray download option – I’ve removed it to show that things do work

Clicking on the menu option displays the following view:

This allows the administrator to select the package to import. Selecting a file and clicking import will copy the package to the server and validate it (it needs a manifest file). However, it doesn’t install it, instead it just tells the administrator about the package:

This allows the administrator to check that the package contains what they actually want! From 1.5, the administrator will be able to install both plug-ins and themes, hence the type label above.

If the administrator is happy, they click on “Yes” and the package is installed. Rather than forcing the administrator to guess about what has happened, a log view is displayed:

The files in gray are extra information – more useful for the developer, rather than administrator – but I’m trying to plan ahead. If there are any warnings or errors, they will also be displayed in this log.

As well as copying the files and changing the configuration, the package loader will also force a reload of the configuration. This gives the following result:

I included the before image in this screenshot, just so it is obvious what has changed. The “Download CCTray” option has been added, and I didn’t even need to restart IIS!

Where To From Here?

At the moment, this is still a very rough prototype. Here are my plans:

I want to do some more work around packages. The process currently has two steps: load the package and then install it. I’m planning on making a packages repository, where the administrator can upload multiple packages. We’ll also pre-install a number of packages to simplify installation of the dashboard.

This repository means the administrator can load packages, and then install or un-install as required. Plus they’ll be able to delete packages from the repository.

Also, as I stated above, I want to make packages reversible. If an administrator decides they don’t want a plug-in or theme in the future, they can go to the repository and un-install the package.

Finally, I want to do some tidying up around the log – it would be nice to hide the developer-only messages by default, but still give the developer the option to view them.

Anyway, these are my thoughts, what do you think of the concept?

Releases, Reviews and Roadmaps

1.4.3 is “In the House”!

This week-end, the 1.4.3 version of CruiseControl.Net was finally released, thanks to Dave’s hard work (currently he’s the only one who can do a release).

The good news is this version offers a lot of minor improvements and bug fixes, so CruiseControl.Net should be more stable. The bad news is there appears to be a couple of areas that have digressed – and Subversion support for externals is one of them.

The full list of what’s new is at http://confluence.public.thoughtworks.org/display/CCNET/CCNet+1.4.3+Release+Notes, but here are some of the highlights:

• There is a new validation tool to check the config file (hopefully this will make it easier to configure CC.Net in future).
• Source control errors are now treated as errors – previously they were ignored!
• Custom icons now work in CCTray. Prior releases had a “broken” implementation that would cause CCTray to crash after a while (and messily).
• Web dashboard can have a custom templates location – which now opens the way for themes in future releases.

On the down side we had a couple of issues with CCNetLive failing after the release. One failure was because of SVN externals – if the trunkUrl property is not set, then the code for checking for externals fails (i.e. throws an exception!) The other issue was around some unit tests failing – this was because some environment variables were being added twice.

Fixes are already in place for both issues, but more works needs to be done around SVN externals.

NDepend Review

Patrick Smacchia has performed a review of the CruiseControl.Net code with NDepend. His full post is available at http://codebetter.com/blogs/patricksmacchia/archive/2009/03/15/analyzing-the-code-base-of-cruisecontrol-net.aspx.

In his post, he not only gives us the statistics and graphs from NDepend, but he also analyses the results to provide a better picture of what is happening. There is some good news, plus some bad news.

First the good news, overall code quality is good – only 1.8% of the methods exceed the classical metrics thresholds, the code is nicely partitioned within fewer assemblies and the code that is unit tested is 80% covered.

Now the bad news – only 37% of the code is covered by unit tests! Plus there are a couple of massive dependency cycles, which increases the complexity of the code. Also, he doesn’t like the “Copy local” option, which I’m not sure I agree with or not.

Now, one of the dependency cycles appears to be in the dashboard, while the other is in the core. Now, we already know the dashboard is complex and needs a massive overhaul (would be nice to move away from our own in-house versions of IoC and MVC), but it was interesting to see we have a similar issue within core. Obviously it’s time to do some tidying up!

Thanks Patrick for your review, we’ll have to see if we can improve on this state, especially with some of the larger features we have for version 1.5.

A Roadmap to the Future

Now that 1.4.3 is out, we’re starting to gear up for the next release. Already we have a large number of new features in the pipeline, plus we’re also hoping to tidy up a lot of the development process.

For the moment we are still working on the road map, but it is possible to see our current plans at http://confluence.public.thoughtworks.org/display/CCNET/Roadmap.

Feel free to send any comments or suggestions to the developer mailing list (http://groups.google.com.ag/group/ccnet-devel). We will also be going through the list of patches in JIRA and see which we can add to the roadmap.

A Very Thorough Guide

I was looking for some ways on how to improve one of my build scripts and I found this series of post by Carel Lotz. He has completely documented the process he went through to set up a CI process, using CruiseControl.Net, MSBuild and a few other tools.

The first post in the series is at http://dotnet.org.za/cjlotz/archive/2007/04/04/part-1-continuous-integration-using-msbuild-cruisecontrol-net-fxcop-nunit-ncover-subversion.aspx.

As an added bonus he has compiled all of the posts together into a PDF document, so you can download it, print it and add your own comments for future reference

Some ASP.NET Magic

Why ASP.NET?

ASP.NET is kind of special. We write an ASP.NET application and set everything up. Then, later on we write some more code, compile it and re-deploy it. Now the magic comes in with the way it is deployed – we can just copy the new DLLs over the old DLLs. And that’s it – no problems, no worries.

Now, try this with a Windows application – ERROR! What happens is we get an error message saying the files are locked and we can’t overwrite them. So what is different, and can we use it in CruiseControl.Net?

The simple answer is YES!

Introducing AppDomains

In a Windows environment the smallest unit of execution is a process (yes, there are threads, but they still require a process to contain them). When a Windows application starts up, it starts a process, which then loads all the code and is away running. Native Win32 applications do have the potential to load and unload libraries, but not managed applications (yes, there are good reasons for it, just don’t ask me).

.NET introduces a new, smaller unit of execution – an AppDomain. An AppDomain is included inside a process, but there can be multiple AppDomains per process. An AppDomain is very similar to a process – the application starts an AppDomain, loads the code and then runs. But, there is one major difference – it is possible to unload an AppDomain without terminating the process.

Unbeknownst to a lot of people, every .NET application uses an AppDomain. When the application starts up, the first thing it does (other than the Win32 start-up stuff) is create an AppDomain and load the starting point (usually Main() or something similar). So, we don’t need to make any changes to use an AppDomain in our code!

The Magic of ASP.NET

Now I’m not an expert on the inner workings of ASP.NET, but here is what I understand of it. When an ASP.NET application starts up, the ASP.NET ISAPI filter starts a new AppDomain for it. This AppDomain has a special property set to allow shadow copying (more on this in a minute). Once the AppDomain is started, it loads the required DLLs (after any compilation) and then starts the ASP.NET framework within the AppDomain.

It also attaches a file watcher to the folder as part of the start-up. When the file watcher triggers because some important files have been changed (I’m not exactly sure what is covered), it fires up a new AppDomain and routes all new requests to that AppDomain. Once all requests on the old AppDomain are finished, it unloads the AppDomain. This provides a nice continuation of service to any clients.

Now, the reason why the libraries are not locked is because of the shadow copy property. When this property is set, the AppDomain doesn’t directly use the DLLs. Instead it first copies them to a cache location, and then uses the copied DLLs. Thus the original DLLs are not locked – the copied ones are instead.

Bringing the Magic to CruiseControl.Net

It would be nice to add “hot swapping” to CruiseControl.Net. This would simplify any deployments and open up a few possibilities for improvements. And using AppDomains it is very easy to do.

This is how I’ve done it.

First, there are two applications in CruiseControl.Net that need this functionality – Console and Service (ccnet.exe and ccservice.exe). These are the hosting applications for Core, Remote and the ancillary libraries (e.g. NetReflector, Log4Net, etc.)

Currently, when these applications start, they directly load the required classes and run them. What I’ve done is separate this functionality and put it into a new class called AppRunner. AppRunner inherits from MarshalByRefObject, so it can make cross-boundary calls (e.g. between AppDomains).

The start-up points in the applications start a new AppDomain and create an instance of AppRunner within the new AppDomain. The new domain has the shadow copy property set, so any assemblies loaded into it will be cached first. The create call returns a reference to the instance (which is actually a proxy reference), so we can call the new instance, in the new AppDomain, from the old AppDomain. The instance in the new AppDomain does all the calls to Core and Remote, so these libraries are only ever loaded into the new AppDomain.

Next, I added a FileSystemWatcher, and set it to monitor changes to ThoughtWorks.CruiseControl.Core.dll (this is the main DLL). When this file changes, it calls Stop() on AppRunner and then unloads the AppDomain. Then it loops back to starting a new AppDomain, creating a new instance of AppRunner, etc.

So, long story short – the Core, Remote and ancillary DLLs are no longer locked. When Core is replaced it will shut down the old AppDomain and start up a new one, thus loading the new DLLs.

Simple

Possibilities

This now opens some future possibilities.

First, it is no longer necessary to shut down the service to deploy a new version of CruiseControl.Net. Instead, just copy over the new DLLs, making sure that ThoughtWorks.CruiseControl.Core.dll is the last one copied. This also allows us to add an automatic deployment process in a farm scenario. In this scenario one server would be the master copy – changing this copy would push the new DLLs out to the other instances (hopefully automatically).

Secondly, we can add some more robustness to the application. If a fatal error occurs in an AppDomain (i.e. something completely unexpected), then we can unload the old AppDomain and fire up a new one. Of course we’d need to be careful we don’t enter an endless loop (e.g. crash, restart, same error occurs and the cycle begins again).

Finally we can add some AppDomain recycling. This is something like the application pool recycling in IIS, where an AppDomain can be unloaded every certain amount of time and then restarted. Again, needs a little bit of planning to make sure we do this in a safe way.

Anyway, these are some ideas I’m thinking of for the future, feel free to let me know if there is any other possibilities I haven’t thought of

Some Small Tweaks

Really Simple Syndication

One of the publishers that is available in CC.Net is an RSS feed generator. This generates an RSS feed that contains the latest build, which includes a list of all the modifications.

Which is ok, but one thing that has always irked me is it only contains the latest build, and no other builds. My RSS reader polls every hour for updates, so if there are multiple builds within the hour (or overnight, week-ends, etc) I lose them!

So I thought I’d tidy up the RSS publisher a little.

Good news

First, the good news. Looking at the publisher it is broken down into a number of methods, each producing a part of the RSS feed. Since it’s been broken down already it’s fairly simple to change just parts of the process, rather than needing to redo the entire process.

Bad news

Now the bad news, it doesn’t use any of the XML libraries in .NET. Instead it custom generates the XML (using strings) and writes them using a StreamWriter. So I can’t just modify parts of the XML generation to include what I want, I’m going to need to do some drastic changes instead.

This also means that there is no allowances for reserved XML characters (&, < and >). Any of these characters in the RSS feed will cause it to break

New news

My end approach is simple, I changed to using XmlDocument. I converted all of the parts that directly manipulated XML in strings to use XmlElements and its various methods. Now the publisher directly reads and writes XML in a safe manner.

With these changes in place, it was easy to expand the publisher to store multiple builds. First the publisher loads the old document (if there is none it creates a new one). It then checks how many builds are already in it, if there are too many builds then it clears space for the latest build. Finally it adds the latest build.

Simple

I’ve also added a configuration option to allow people to specify how many builds they want to record. The syntax is:

  1: <rss items="100"/>

By default the value is 20. If people want to go back to the previous functionality (i.e. only the latest build) then all they need to do is set this value to 1.

Hopefully this will make RSS feeds more useful.