Playing with Packages

Still Downtime

We’re still waiting on the 1.4.3 release, so I thought I’d try and implement another of my ideas. At the moment the implementation isn’t perfect – I know it needs some fine-tuning – but it provides another interesting possibility for what we can do with CruiseControl.Net.

Packages

I spent three years working for a university here in New Zealand, so I’ve had some experience with content management in this field. One of the concepts that’s been around for a while is the concept of a package – a set of files that are related, plus a manifest describing these files (and how they can be used). These packages are normally stored as a ZIP file to allow ease of transport.

The reason for a package is to enable blocks of content to be passed around easily. You’re setting up a new course, and discover that someone else has already written a module on one of your topics. Rather than re-writing the whole module, you just import the content package into your Learning Management System.

As you can see packages simplify knowledge transfer and make it easier for people to use it. Now in CruiseControl.Net we’re not interested in learning content (or at least most of us aren’t), but we do have build artefacts – the results of a build process. It would be nice to have some simple and easy way to transfer these around.

Packages for CruiseControl.Net

I thought I’d implement packages for CruiseControl.Net. A package can be defined as a group of files in a ZIP archive, that may or may not include a package manifest. The manifest lists all the files, plus contains a header detailing the build that generated the manifest (this is an area that probably needs a bit more work).

Since I wanted to give people control over what is packaged, I have added a package publisher. This is an example of how a package is defined:

  1: <package name="Binaries" manifest="true">

  2:   <files>

  3:     <file>deploy\*.sql</file>

  4:     <file>deploy\*.msi</file>

  5:   </files>

  6: </package>

The key attribute is the name – this will be the name of the package when it is generated. I’ve made the package so the manifest is optional, and this example shows how to turn it on.

The other important part of the definition is the files element. This defines which files should be included in the package. In the example I’ve shown how a wild card can be used for the file list, additionally individual files can also be specified.

There are a few more options for the publisher. The full list is:

Name	Type	Description
name	string (required)	The name of the package.
compression	integer (optional)	The compression level to use. This must be a value from 0-9, where 9 is the highest compression. The default value is 5.
always	boolean (optional)	This records whether a package should always be generated. If this is off, then a package is only generated for successful builds. If this is on, then a package is generated whether the build succeeds or not. The default value is false.
flatten	boolean (optional)	Whether to flatten the file structure in the package or not. If this flag is off then the directory structure is also included, otherwise just the file names are included. The default value is false.
single	boolean (optional)	Whether a package should be generated for each build, or just one package for the project. If this flag is off, then a package file is generated for each build and stored in the artefacts folder under a new folder for the build. This is useful for maintaining a history of the packages, but can result in a large number of packages accumulating. If this flag is on, then only one package will be generated. When the package is published, it overwrites the previous package. The default value is false.
fromWorking	boolean (optional)	Sets whether to get the source files from the working folder (true) or the artefacts folder (false). The default value is false.
manifest	boolean (optional)	Whether to include a manifest file (true) or not (false). The default value is false.
files	string array (required)	The files to include.

I’ve tried to cover all the most obvious options, but like I said earlier, it still needs a bit of polish.

The following shows the package that is generated in this example:

Since I didn’t set the “single” option, it will be created in its own folder underneath the artefact folder.

Viewing Packages

Generating the packages is only half the group – the other half is to make these packages easier available to the users of the system.

To achieve this there are two places these packages need to be: in the dashboard and in CCTray. As such I’ve made a couple of changes to ICruiseManager to allow transmitting files (I could have done it via a shared folder, but that always assumes folder will be available, which might not be the case over the Internet). I’ve also added a method to retrieve a list of files. I’ll go into the technical details in a future post, but here are some shots of how it works.

First, the package list is available from CCTray by right-clicking on a project:

I have added a menu option called “Packages”, plus a short-cut key for it. Clicking on this option will bring up the list of packages:

This contains some obvious data (name, date and size), plus the build that the package is from. Double-clicking a package, or clicking on the download button (the little globe with an arrow) will first prompt the user for a download location, and then download the package.

The dashboard also allows downloads of packages, although in a slightly different way. Instead the list of packages is included on the project status, and are downloaded by clicking on the link.

Under the Hood

As a final note, I’ve implemented this functionality using #ZipLib from the #Develop team. It works nicely and without any issues.

The transfer of files across .NET Remoting is a little, since passing a stream is not recommended. Instead a stream “rip-off”, that provides similar functionality. This means it’s up to the client to control how data comes across.

Finally, the storage of the files is just in the file system. The more I play around with CruiseControl.Net, the more I question it’s reliance on the file system for all storage. It would be nice to have a centralised data store that could be accessed by the different parts without going through either the file system or the server, but I guess this is a topic to look at in future.

Where’s the Code?

At the moment this code isn’t publicly available. The reason why is simple: I started playing around with this before the change freeze (although I didn’t get far), and as such it’s based on the trunk code. Now that the change freeze is in place, I can’t deploy it until the release

So until the change freeze is lifted, this code will have to wait…

In meanwhile, I’ll look at writing a post that dissects how this functionality works. That way i won’t forget how it works either!

Simplifying Communications, Part 2

The Story Continues, Again…

In my last post (here) I modified the server in preparation for building a communications client. This involved adding a new method to handle “generic” messages. This method took in an action name and a request message, and then returns a response.

Now it’s time to build on this and start putting together a communications client.

New vs. Old

The first question is where will the communications code go? At the moment there are two choices – Remote or a new project.

Remote is currently the communications library within CruiseControl.Net, but it also handles a number of other items (e.g. exceptions, events, interfaces, etc.) Additionally it depends on the NetReflector library.

Adding a new project would mean potential duplication, although it would be smaller!

In the end I decided to add a new project in the same folder and link to the existing files. The new project has the same namespace (although a different assembly name). Additionally I modified a couple of files to include compiler directives (#if…#endif) to exclude the NetReflector classes – this removes the dependency on NetReflector.

Now I can use the same classes, but without all the extra baggage. Additionally I can include the new communications classes in Remote later on if I think they are necessary for the main server.

The Design

First off, I want a generic interface to the communications. This will be a single class that handles all communications calls – this way the application does not need to know whether it is HTTP, .NET Remoting or something else.

Secondly, there will be a number of “transport” channels. Each one of these channels contains the actual communications details (e.g. HTTP, .NET Remoting, etc). When the application wants a new client, it instantiates the channel, start a new client and links the two together.

Finally, to simplify the initial set-up, there will be a factory class to do the above step – plus handle some extra steps (auto-detection of the protocol, etc.)

The following diagram shows the various components:

Implementation

The code consists of five classes and one interface. These are:

• IServerConnection
• RemotingConnection
• HttpConnection
• CruiseServerClient
• CommunicationsException
• CruiseServerClientFactory

Most of these should be straight-forward

IServerConnection

IServerConnection defines the channel to the client. It has one method – SendMessage() and one property – ServerName. ServerName is the name of the server that the connection is for. This is used to set the Server property on the request (since this required for web connections). SendMessage() does the actual sending of the message. The input arguments are the action and a ServerRequest, and it returns a Response.

This is all it does, it doesn’t need to perform any validation of the response, as this will be handled by the client.

RemotingConnection and HttpConnection

These are two implementations of IServerConnection – one for .Net Remoting and the other for HTTP. They handle connecting to the server, converting the messages into wire formats and then converting back again.

RemotingConnection is very simple – the only fancy logic in it is generating the URI for the remote server (and even so, is still very simple).

HttpConnection does a bit more work as it converts the request message to XML and then has to convert the response from XML. Internally it uses a WebClient instance, and just calls UploadValues() to send the data (this is a POST request).

CruiseServerClient

This is the biggest of the classes. Each available action on the server has a matching method here. CruiseServerClient takes the incoming arguments and puts them into a request message (of the correct type). This message then gets passed onto the IServerConnection instance, which sends it to the server and returns a response.

When the response is returned CruiseServerClient validates the response. If the response has a status of failed, it will throw an exception (see CommunicationsException below), otherwise it will return the data items (if any).

This class has similar helper methods to what was added to CruiseManager – GetServerRequest(), GetProjectRequest() and ValidateResponse() – to help with this processing.

It also allows the calling application to set a target server. If a target server is not set, then the server name from the server connection is used. Otherwise the target name is used. This is required for HTTP calls where the target server and the connection server can be different.

CommunicationsException

A CommunicationsException is thrown whenever the remote server returns a failure response. This encapsulates the errors and makes it easier for the calling application to differentiate between errors from the remote server and errors on the client side.

CruiseServerClientFactory

The final class is CruiseServerClientFactory. This provides helper methods for starting a new client. Currently there are helper methods for starting a .NET Remoting connection, an HTTP connection and a generic connection.

The generic connection will attempt to detect the connection type from the URI. If the URI starts with http://, then an HTTP connection is started. If the URI starts with tcp:// then a .NET Remoting connection is started. Anything else throws an exception.

This class can be bypassed and the calling application can start its own IServerConnection and wire it to a CruiseServerClient. But this class will help reduce coding in most situations.

An Example

Now that I’ve built the library, now is it used?

First, add the assembly ThoughtWorks.CruiseControl.Communications to the calling project. Just remember that the namespace for this library is actually ThoughtWorks.CruiseControl.Remote – not Communications.

Then instantiate a client. The following shows a few different approaches:

// Generate an HTTP client that targets another server
CruiseServerClient client = CruiseServerClientFactory
 .GenerateHttpClient("http://buildserver/ccnet", "actualserver");
// Generate a .NET remoting client explicity
CruiseServerClient client = CruiseServerClientFactory
 .GenerateRemotingClient("buildserver");
// Generate a .NET remoting client implicity (by auto-detection)
CruiseServerClient client = CruiseServerClientFactory
 .GenerateClient("tcp://buildserver:21234");

To then call the server is as simple as the following:

client.ForceBuild("ProjectName");

And that’s it. Any errors will be raised as exceptions, so make sure there is a try-catch block somewhere around the code.

Where To From Here?

What I have done so far is the basic version of the communications client, which covers most of what the current code can do. Now it’s time to start expanding it to cover other scenarios, e.g. security, handling proxies, other communications channels, etc.

Also, I haven’t covered it in this post, but I have updated my CCCmd (command-line remote client) interface, so next step is to document it and include it in the main code base. I’ll also look at CCTray, the dashboard and the triggers and publishers that connect to other servers.

Another area is to make custom versions of this library for different platforms – e.g. Silverlight, Compact framework, etc. However for Silverlight it needs an asynchronous model, and I haven’t figured out what is the best way to do that yet.

Finally, I’m toying with the idea of making the communications library backwards compatible (e.g able to connect to 1.4.3 servers or earlier). Unfortunately this will require a lot more work as the older versions don’t have the generic message handler So this one might go in the too hard basket.

Throttling Build Integrations

My Server Can’t Handle It

Recently on the mailing group there was a request to limit the number of simultaneous build integrations. Well, the person actually was asking how to get an integration queue to fire off more than one request at a time, but the underlying intention was to limit the number of integrations. Now playing around with integration queues can be a risky thing (just trying to figure them out can be challenging!) so I thought I’d use the new integration events to add a build integration request throttler.

The principal behind the throttler is simple – it will limit the total number of concurrent builds on a server. This is completely independent of queues, so both queues and build throttling can be active on a server. Since not everyone will want this functionality, it must be optional, and it also should allow people to set the number of concurrent builds allowed.

Right, that’s the design brief, let’s start building it.

Extension Framework

First, a quick review of server extensions. These extensions are designed to plug in to CruiseServer and allow direct manipulation of CruiseServer (i.e. they can call any methods on it). They can also subscribe to a number of events that have been added to ICruiseServer, which now includes events for before and after an integration request is triggered.

All extensions must implement ICruiseServerExtension, which provides the following methods:

• Initialise()
• Start()
• Stop()
• Abort()

Initialise() is called when the extension is first loaded (i.e. when CruiseServer starts up) and can be used to subscribe to any events. Start(), Stop() and Abort() are called by the matching methods on CruiseServer and allow the extension to do processing at any of these points.

The Throttler Extension

I have added a new extension to the Core project called IntegrationRequestThrottleExtension. This implements the four methods required by ICruiseServerExtension – but it ignores the last three methods (i.e. does nothing).

The Initialise() method checks the configuration to see if the number of requests allowed is set (more on this later) and then subscribes to the IntegrationStarted and IntegrationCompleted events. So far, so good.

In terms of the actual throttling, I store a list of project names within the extension. When a new request is started the project name gets added to the list. If the list is over the limit, then the extension sets the result code to Delay, otherwise it sets the result to Continue. Likewise, when a request is completed it is removed from the list.

Since the server is multi-threaded I needed to add some locking code – otherwise the extension would get garbled data. Additionally, requests must be processed in order – so I added some code to prevent requests “jumping” the queue.

And with that throttling is now available for CruiseControl.Net.

Configuration

Now that the extension has been added, the next question is how to use it? The answer to this lies in configuration. In the application configuration (ccnet.exe.config or ccservice.exe.config) there is a <cruiseServer> section. The extensions need to be added to here.

The following is an example of how to add the throttling extension with a limit of three requests:

  1: <cruiseServer>
  2:   <extension type="ThoughtWorks.CruiseControl.Core.ServerExtensions.IntegrationRequestThrottleExtension,ThoughtWorks.CruiseControl.Core">
  3:     <limit>3</limit>
  4:   </extension>
  5: </cruiseServer>

The default limit is five – so if the <limit> element is omitted (it is optional) then only five can be run concurrently.

Final Note

At the moment the code for this extension (and the extensions framework) is all in the security branch. This is because we’re still waiting on the final release for 1.4 (1.4.3 is still being delayed!) and all this new functionality will be part of the 1.5 release.

New Integration Events

Welcome to the Rabbit Hole

Recently I modified CruiseControl.Net to allow adding server extensions. These extensions allow an external developer to modify how the core server works (initially I added it to allow WCF communications). Later I modified the server again to raise events when certain user-triggered actions occurred (e.g. force/abort build, start/stop project, etc.) These now allow extensions a view on what is happening with the server.

But, there is one set of events that I hadn’t added – the ability for an extension to see when an integration starts or stops. Since I have a little bit of time on my hands, I thought I’d take a quick break from messaging and implement this.

Now, in case you were wondering about the rabbit hole – do you want the blue pill or the red pill?

Layers Upon Layers

While this may seem like some nice easy functionality to add, it’s not. The reason – integrations are handled by ProjectIntegrator, events are fired by CruiseServer – and the two types know nothing about each other! In fact, there are many layers separating the two classes.

After delving into the code, there are the following layers between the two classes:

• CruiseServer
• IntegrationQueueManagerFactory
• IntegrationQueueManager
• ProjectIntegratorListFactory
• ProjectIntegrator

So, in order to get the two classes talking to each other we need to go through all these layers!

In the end I decided that the easiest way was to pass an instance of ICruiseServer through all these layers and store it in the ProjectIntegrator. Now the integrator knows about the server and can call methods on it.

Adding the Events

To handle the events I added two new classes:

• IntegrationStartedEventArgs: This contains the IntegrationRequest request and allows the extension to cancel/delay/continue the request.
• IntegrationCompletedEventArgs: This contains the original IntegrationRequest request and the outcome of the integration.

I wanted to include IntegrationResult in IntegrationCompletedEventArgs, but IntegrationCompletedEventArgs is in the Remote project (and needs to be since it’s an event on ICruiseServer) while IntegrationResult is in Core. Therefore I could only put in IntegrationStatus To move IntegrationResult to Remote would take a lot of work and I’m not sure what risks this would open.

The actual events are on ICruiseServer and are called IntegrationStarted and IntegrationCompleted. Finally I added a couple of methods to actually fire these events, since the actual firing will be done by ProjectIntegrator. These events are implemented in CruiseServerEventsBase.

Firing the Events

Now that I’ve done all this background work, I’m finally ready to fire the events

The firing is done in ProjectIntegrator in the Integrate() method. When a request is available from the queue it calls FireIntegrationStarted() and checks the result.

If the result is continue, then the standard processing continues and FireIntegrationCompleted() is called at the end.

If the result is delay, then ProjectIntegrator enters a loop. It calls FireIntegrationStarted() every 200ms (is this too short?) until it returns a result other than delay.

Finally, if the result is cancel, the request is removed from the queue without processing.

End of Change

Very simple, but now the extensions have the ability to modify the integration behaviour of the server.

In my next post I’m going to test these new events and add an extension that will throttle the number of concurrent builds

Stay tuned…