Holiday Plans

Yes, It’s That Time Again!

Once again we are coming up to our annual trip to China to see the in-laws. This means I will be disconnected for the next three weeks, but I will have plenty of time on my hands. Last year I worked on a replacement for CCTray (Project Leo) which later became FastForward.NET. This year my plans are just as grand

Currently I am working on converting CruiseControl.NET to use streams instead of strings. Given that this is a big task, with lots of complexity, it’s been slow going. I’m aiming to finish this in my time away – plus look into some other possibilities for the concepts (like distributed builds).

Second, I’m looking at building a Silverlight front-end for CruiseControl.NET. I’ve been playing around with Silverlight at work, so this opens a few possibilities. This will be a standard plug-in for the dashboard, so people can choose to use it or not if they desire. The only downside is it will require .NET 3.5 and Silverlight 3 – so I’m not sure how well this will work on *nix environments.

So, you won’t hear much from me for a wee while, but I hope to have some exciting stuff soon!! Plus my Cantonese should be better by the time I return

One Year On

Happy anniversary! Yes, my first post on this blog was one year ago! That was when I first adventured into the wonderful world of open source and embarked on enhancing and improving CruiseControl.NET.

Since then a lot of things have changed! Here are some of the highlights:

• I have a son (he’s almost five months old now!)
• I’ve written 178 posts
• I’ve changed 60 thousand lines of code (wow!)
• I’ve added a new application to CC.NET (CCValidator)
• And I’ve started my own open source project (FastForward.NET)

In terms of CruiseControl.NET, I’ve added the following functionality:

• Security
• Dynamic build values and parameters
• Message-based communications
• Common communications client
• Monitor API
• NDepend and NCover tasks
• Image and HTML support in the dashboard
• Packages
• Hot-swapping
• Parallel and sequential tasks
• Task statuses

Plus I’ve gotten to know a great bunch of people, and learnt a lot about coding and development.

So, all in all, one year down the line, I think it has been very much worth it

Here’s to another year!!!

FastForward.NET Information

I’m still in the process of getting FastForward.NET off the ground. One of the really great things about working on CruiseControl.NET is most of the infrastructure is in place. Unfortunately, starting a new Open Source project means I have to set it all up myself

That’s the bad news, the good news is I am slowly getting there. Here is some project-related information:

Source Control: this is being hosted at SourceForge for the moment. I know SourceForge is slow, but I’m hoping they’ll improve. Here is the project site: https://sourceforge.net/projects/fastforwardnet.

Issue Tracking and Documentation: again, I’m using SourceForge – I am using the hosted version of Trac. This has some nice features, plus all the information is in one place. What I really like is the roadmap section – I’ve started setting things up for how I plan on moving forward. The only down side is it is slow, again I’m hoping the SourceForge will improve. The location is https://sourceforge.net/apps/trac/fastforwardnet/.

Finally, I’ve set up a Google mailing group at http://groups.google.com.ag/group/fastforwardnet. I imagine this will be similar to what we have for CruiseControl.NET – general discussions will occur here (including help if necessary) with issues being raised on Trac as they come up.

Also, I hope to be able to use a CruiseControl.NET soon, in which case the project will be under continuous integration. Unfortunately I don’t have a server available, so I’ll need to convince some kind-hearted person, so stay tuned on this front…

Finally, if anyone wants to submit code for this project, I’m am more than willing to grant access to Subversion. The project is under a MIT/X11 license, so it’s very, VERY free. Just don’t wreck the code

Anyway, that’s enough for now, let me know if there is anything else you want to know.

CruiseControl.NET and Ohloh

Introducing Ohloh

Ohloh is a social site for open-source development – it allows developers to track what is happening with their projects, to “toot their horns” so to speak. Their website is at http://www.ohloh.net/ – and you can even see CruiseControl.NET on there (https://www.ohloh.net/p/cruisecontrol).

Some More Stats

Since we are both open source projects, I thought I’d add a quick link between the two. This link means you can now see Ohloh stats for your open source project within CruiseControl.NET itself.

Actually, I didn’t think of this idea myself – instead I got it from the Subtext instance of CruiseControl.NET (http://build.subtextproject.com/ccnet/ViewFarmReport.aspx). This is another open source project – one that uses both Ohloh and CruiseControl.NET. Actually they’ve done a very nice job of customising CruiseControl.NET – take a look sometime.

Rather than using their approach, I’ve taken a more limited approach (their Ohloh stats are displayed on the main pages – which is ok for them since they have CruiseControl.NET for only one project). Instead I’ve added a plug-in that enables a project to be linked to Ohloh.

Here are some pictures of what I’ve implemented:

This first picture shows the Ohloh quick summary next to the project title. Also, there is now a “View Ohloh" Stats” menu command (on the left). Clicking on this link brings up the Ohloh widgets in their own page:

Configuring this requires two steps:

Configure the server

since the stats are on a per-project basis the actual link is stored in ccnet.config. To configure a project to link through to Ohloh, add the following configuration:

<linkedSites>

  <namedValue name="ohloh" value="322"/>

</linkedSites>

In this case the project id for CruiseControl.NET in Ohloh is 322. This would be replaced with the identifier of the project.

Add the Ohloh Stats Package

The first step configured the server, this step configures the plug-in in the dashboard. The quick summary will be enabled when the Ohloh link is added, to add the “View Obloh Stats” command this plug-in must be installed.

In Conclusion

This was a nice simple feature to implement (for a change!) If you have an open source project, give it a spin and let me know what you think (sorry – this is only for open source).

A Son is Born

This morning at 6:25am, my son, Ethan Philip Sutherland, was finally born. He is a nice healthy young boy, although certainly on the heavy side (especially considering the size of my wife!)

Normally, I aim not to post personal events in this blog, but this event will have a significant impact on my life and the various tasks I do, hence the reason for the post.

I am taking the next four weeks off work to help my wife adapt from a two-person to a three-person family (an yeah, I need the time too!) As such, I probably won’t have much time to work on CruiseControl.Net either, although this will be even more impacted.

I had hoped to have the security branch merged and tested by this time, but due to continuing delays in completing the 1.4.x releases, this hasn’t happened The code is 95% complete for the following features:

• Security
• Dynamic build parameters
• Messaging
• Communications client
• Server extension points
• Build throttling

I say 95% completed, because there has not been much feedback or testing on these new features. As such, I know there will be additional issues or items that I have not thought of. So, these items are on hold indefinitely, as I have no idea of when I will have time to do them. Perhaps one of the other developers will take these features and complete them…

Additionally, there are a few other items I’ve been working on which haven’t been checked into Subversion anywhere. These include the following features:

• Simplified web administration
• Installation packages
• Detailed task statuses for builds
• Package publisher
• NDepend task (plus installation package)
• XMPP publisher
• Dashboard themes
• Extended source control exception handling

These are in various states of completion, depending on where I got these to. Most of these are proofs-of-concepts, so I only developed them to the point where I was happy the concept was valid. The idea was to then commit these into the trunk and finish them.

If I have time, I may add some of these items in, but then it will make the lead developer unhappy as we are only supposed to be bug-fixing.

Finally the VS2008 migration is also on hold. This work has been completed and tested, but thanks to the on-going issues with 1.4.x cannot be merged!

So, you won’t see many posts from me for the next while, hopefully I will have time to work on CruiseControl.Net, but I’m not planning on it for a while.

Thanks for all your support,

A proud (but tired) new dad

Adventures with NetReflector

Introducing an Old Project

When I started developing for CruiseControl.Net I came across an external library called Exortech.NetReflector. This library is responsible for the de-serialisation of the configuration within core (and the dashboard) was was originally developed by the same guy who originally wrote CruiseControl.Net – Owen Rogers.

In a nutshell NetReflector allows dynamic de-serialisation of XML. While the standard XML serialisation attributes in the CLR are pretty good they don’t allow for the scenario where the actual de-serialisation types are unknown at compilation time. NetReflector allows for this scenario.

Now, while NetReflector is very stable (there have been no modifications to it for a long time) and does its job very well, there were a couple of enhancements I wanted to add.

So to cut a long story short, I asked Owen for access to modify the project and he has given me the keys to the entire project (yes he made me an admin, even though I only asked for dev rights). So, every now and then I will also be posting on enhancements to NetReflector.

First Enhancements

My first enhancements for NetReflector are to make the validation of CruiseControl.Net configuration easier. Sometimes the failure messages from NetReflector are downright cryptic (Object reference not set – where and why?)

This can make writing or modifying configuration files harder, especially if someone is new to CruiseControl.Net. Additionally, trying to support some of these issues often means trying the same thing and trying to work out where the failure is (which for me is often trial-and-error).

Initially I’m modifying the following errors:

Error	Enhanced Error
Invalid item when loading an array item.	Will now tell the user which item from which array and include the faulty XML.
Null reference error when the type attribute isn’t set.	Will now tell the user that NetReflector can’t handle the item. If a type attribute is required it will tell the user that the attribute is missing. The faulty XML is also included.
Attributes allowed on base types (e.g. strings).	This will now throw an exception telling the user attributes are not allowed.
Sometimes hard to work out where missing elements should be.	Now includes the parent element so the user can see where the element is missing from.

I’ll make a new build with these enhancements and add it to CC.Net sometime.

Wix and CruiseControl.Net

An Introduction to Wix

I guess the first thing I should do in this post is introduce Wix. For those who haven’t met Wix before it stands for Windows Installer XML. Basically it’s another open source tool for building MSI packages. The main difference between this and some of the other tools (e.g. Nullsoft) is it is entirely XML-based. Since MSI packages are based on internal tables, not setup scripts I think the two go well together.

Now I tried to take a quick look at the existing installer in CruiseControl.Net, but I couldn’t find where it is. Since I took the easy approach and gave up and changed to something that I’m used – Wix.

A Newer Installer

For those of you who are wondering why I’m doing this in the first place, I wanted to make it easy for people to try my new extensions to CruiseControl.Net (WCF and security). The idea being if I can make it easy for people to install, then hopefully people will start using it and give me some feedback on how I can improve it.

So, what is in my new installer. First, there are the standard components:

• The console and service versions of CruiseControl.Net (the actual heart of the system)
• CCTray
• Web dashboard

Next I’ve added my WCF extensions:

• WCF extension for the server
• WCF custom transport for CCTray
• WCF router

Finally I added in CCCmd (my command-line interface to CCTray) to allow people to use non-UI tools (e.g. scripting languages, etc.)

In case you’re wondering where the security is, it’s been built into the standard components. People can turn off security, but it cannot be removed. In future releases of CruiseControl.Net it will be in the main product line, but I don’t consider it ready yet to be added into the trunk.

I’ve also spruced up the interface a little – I’ve given a new logo and splash banners in the installer. I’ve also included the standard license. People have a choice of typical (just the standard components), custom or full installs – so they can choose exactly what they want.

I’ve got the service installation working, but I’m having problems with the IIS registration (not sure if it’s my machine or something else).

Where is it?

For the moment I’ve added the Wix code to the security trunk (we can discuss what to do with it in the long term later). I have also loaded it to my google repository (here) so it will be available for general access.

Please let me know any feedback on it, or on any of the WCF/security work. I should mention that this is a work in progress – I hope to smarten it up later on and add some nice functionality to it (more on that to come).

CruiseControl.Net Goodies

Who Likes Configuring CC.Net?

Most of us at one time or another have have problems with configuring CruiseControl.Net. The server configuration is based on an XML config file, and get get rather verbose at times! Now while ThoughtWorks does provide pretty good documentation on the process (read it here), I still have problems at times.

On a recent post to the CC.Net user mailing group somebody asked if there was an up-to-date XML schema that we could use with intellisense in Visual Studio. He even mentioned a location of a schema – the only problem was on investigation that was the schema for CruiseControl – not CruiseControl.Net. So I went looking. I did find a schema in the code – but it doesn’t look very complete. Then, I found a nice little tool – CCNetConfig.

CCNetConfig is a graphical editor for CruiseControl.Net config files. Now I’m not sure who the main developer for it is (his user name is camalot), but it is another open source project – this time hosted at CodePlex. The project itself is hosted at CodePlex – http://www.codeplex.com/ccnetconfig.From the looks of things it is still be worked on – another not much work appears to be happening at the moment.

I haven’t played with it yet, but from the screenshots it looks nice! I’m downloading the latest build now and I’ll try and put it through it’s paces.

Community Plugins

A related find is also CC.Net Community Plugins – another open source project hosted at CodePlex (http://www.codeplex.com/ccnetplugins). These are various plug-ins that different people have written to work with CruiseControl.Net. They list the following plug-ins:# Labellers

• LastChangeVersionLabeller
• TfsWorkItemPublisher
• CodePlexReleasePublisher
• RssBuildsPublisher
• MetaWeblogPublisher
• TwitterPublisher
• PowncePublisher
• FtpSourceControl
• MbUnitTask
• XUnitTask
• NCoverTask
• BuildPublisherCleanupTask
• FxCopTask
• MsTestTask

Plus this is an interesting component called The Macro Engine which looks like it can dynamically modify parts of the configuration on the fly.

Again, I haven’t played with any of these, but they might come in useful later on

An Overview of the WCF Extensions

Introduction

Recently I documented adding a WCF extension to CruiseControl.Net. Due to mono-requirements I had to do it in a way that I wouldn’t force an upgrade of the CruiseControl.Net source code to .Net 3.0 or 3.5. Here is the complete set of posts:

• CruiseControl.Net and WCF
• WCF Server Extension
• Cruise Control Contract
• Transporting Data via Data Contracts
• WCF Router
• Extending CCTray
• Configuring a CCTray Custom Protocol
• WCF Custom Protocol

This will be my last post in this series as I’m preparing to move onto some more work (adding some security extensions to CruiseControl.Net). However I thought I ought to add this final post just to wrap up what I have done.

Before WCF

Before I started adding the extensions CruiseControl.Net consisted of three main components:

• Server (whether as a service or running as the console app)
• Dash board
• CCTray (the client application)

These communicated in the following ways:

Initial CC.Net communcations

As you see .Net Remoting was the only way to communicate with the server, thus the client either had to use .Net Remoting or go via the dashboard.

Extension Points

The first change was to add some extension points to both the server and CCTray. This would allow me to add the WCF code without converting the rest of the application. With the extension points the system now looked like this:

Extension points to CruiseControl.Net

I didn’t add any extension points to the dashboard as I’m planning on going directly from the server (the worker of the system) to CCTray.

WCF Extensions

WCF Extensions

Now that I had some extension points for communications, I added the actual WCF components. These consisted of a server-side WCF extension, plus a custom transport protocol for CCTray.

The server side extension used the new hooks in CruiseServer to start and stop/abort a WCF ServiceHost. I added a service contract, ICruiseControlContract, plus its implementation and a number of data contracts to allow data extract. The only thing that a server administrator now needs to do is configure the server to use WCF – which uses the standard WCF configuration for the following service/contract combination:

• Service: ThoughtWorks.CruiseControl.WcfExtension.CruiseControlImplementation
• Contract: ThoughtWorks.CruiseControl.WcfExtension.ICruiseControlContract

CCTray is a custom transport extension. I added transport extensions so other people can also build their own transport mechanisms if desired. The main points to a transport extension is they must provide implementations of ICruiseProjectManager and ICruiseServerManager. These implementations do most of the actual work in CCTray, and they are returned from factory methods in ITransportExtension. For the WCF extension I implemented these classes and passed on all the calls to the client code generated by svcutil.

There is no need to modify the app.config for CCTray as the bindings and endpoints are generated in the code from user input – making it even simplier for end-users to set it up on their machines.

Routing Messages

The final piece of the puzzle is a router to sit in IIS:

WCF Extensions plus Router

This is required for installations that already have IIS listening on port 80 and want to use HTTP transport within WCF. Without the router we would have to use a different port to listen for requests.

The router merely needs to be added to IIS (either in an existing .Net 3.0 application or in its own application) and then its web.config modified to point to the listening WCF extension (which can listen on any port or protocol it likes).

What’s Next?

That, in a nutshell, is the work I did to allow WCF to be used in CruiseControl.Net. The extension points are general and can be used for other, non-WCF extensions.

Now I haven’t quite finished the work on the WCF extensions – they are missing one important component. Currently I am not passing any user information around! This is a major issue as ForceBuild(), AbortBuild() and FixBuild() all need this information. However I haven’t figured out an easy, secure way of passing this information, so I’m going to leave it for the moment.

At the moment I’m working on fixing a few issues that have been raised in Jira to try and get a better understanding of the code. Once I have finished these I’m going to look into adding a security layer to CruiseControl.Net that is easy to configure, flexible to expand and yet powerful enough to handle any security scenarios.

CruiseControl.Net and WCF

CruiseControl.Net

I’ve been using CruiseControl.Net for years. As a Continuous Integration (CI) server it is without peer in the open source community (heck I even think it’s better than some of the non-open source versions). I like it so much that when I changed to my new job a year back that first thing I did was set up a CruiseControl.Net server.

However as much as I like the application there are a few things that bug me. One is the dependence on .Net Remoting for communications between the client (CCTray) and the server. I know it’s possible to use an HTTP transport but I’ve never had any success in getting it to work as nicely as the Remoting transport.

Until my new job I was able to live with it. But at my new job we use a Virtual Private Network (VPN) for remote access, and sometimes it’s nice to work from home. The down side is the VPN only allows certain ports – everything else gets blocked. And surprise, surprise the port that remoting uses is blocked. I could swap to port 80 but then I won’t be able to use the web sites on our build machine (it also hosts a number of other development tools like our source control and bug tracking applications), plus I think the web dashboard is critical for investigating why the build failed.

So it’s time to delve into the world of Open Source and get involved in improving an application. Initially I’m planning on adding WCF to CruiseControl.Net, but I also have a few other ideas to add in (more on them later).

Welcome to Open Source

I read through the instructions on getting involved in development and one of the things it recommended was sending out development ideas to the group of developers involved. So I joined the mailing groups and asked of their opinion on WCF. Overall the feedback was that it sounds liked a good idea – with one caveat. This was it needs to work in Linux under Mono – which doesn’t implement WCF (yes I know there is the Olive project, but it doesn’t seem like much is happening with it). So any development on adding WCF to CruiseControl.Net mustn’t break compatibility with .Net 2.0 (which added some challenges considering WCF is part of .Net 3.0 and later).

To get around this what I’ve been doing is making the WCF components as extension components to CruiseControl.Net and CCTray. The first part is actually adding the extension points to both the server and CCTray (neither of them has the deep integration points that I’m after).

The Nuts and Bolts

My initial work has involved adding some extension points to the server and testing whether it is possible to expose some functionality as a WCF service. As I’m wanting the WCF service to start and stop when the server does I’ve added an interface called ICruiseServerExtension. This exposes four methods:

1. Initialise()
2. Start()
3. Stop()
4. Abort()

Start(), Stop() and Abort() are called by the server when it starts, stops or aborts (simple enough). Initialise() is called when the extension is first loaded. To do this I have added a new method to CruiseServer called InitialiseExtensions(), plus added an extra parameter to its constructor. The extra parameter is simply a List<ICruiseServerExtension> of all the extensions to load (more on how this list is generated later). If there are extensions in the parameter then InitialiseExtensions() is called. This performs the actual loading of the extension object then calls initialise and passes in the CruiseServer instance plus the configuration for the extension.

The list of extensions to load is stored in the app.config file for the server. When CruiseServerFactory performs the creation of a new local ICruiseServer it also loaded the definitions from the config file. I added ServerConfigurationHandler to handle the parsing of the config and return the list of extensions.

That pretty much explains the new extensions to the server side of CruiseControl.Net to allow WCF without forcing everyone to use .Net 3.0 or later. In my next post I’ll talk about the actual WCF extension and how it works.