Distributed Builds: Part 4 (of 4+)

Quick Recap

This is the fourth in my series on adding distributed builds to CruiseControl.NET. Previously I’ve covered some of the issues, a basic design and some scenarios that need to be covered.

In this post I’ll start to look at how we can possibly configure a distributed build scenario.

Starting Off Easy

Of the six scenarios in my last post (here) the simplest one to implement is the single remote build scenario (well, the local build scenario is easier, but then nothing is distributed!) So I’ll look at how I can configure a project to be distributed.

NOTE: this is a work in progress, it is only my current ideas of how to implement it. Hopefully with feedback we will be able to improve it over time.

Machine Definitions

The configuration needs two sections. First we need to define some remote machines. These definitions provide the connection and metadata information about the remote machine. For example, we can define the following machines:

<remoteMachine name="buildAgent1">
  <address>http://winMachine</address>
  <configuration>
    <namedValue name="OS" value="Windows7" />
    <namedValue name="Browser" value="IE8" />
    <namedValue name="Browser" value="Firefox" />
  </configuration>
</remoteMachine>
<remoteMachine name="buildAgent2">
  <address>http://linuxMachine</address>
  <configuration>
    <namedValue name="OS" value="Ubuntu" />
    <namedValue name="Browser" value="Firefox" />
  </configuration>
</remoteMachine>

buildAgent1 has a target address of http://winMachine (at some future point we need to do some more work around the addresses, but it will do for now.) When a project builds it will send the request to this address. It also has some configuration values – it defines an OS and two browsers. buildAgent2 is very similar, but it has a different address and configuration values.

The address element will be mandatory (after all it is very hard to send a request to the machine if its address is unknown!) while the configuration element will be optional. The configuration element is basically to provide hints to the project on which machines it will choose.

These elements will be top-level elements in the configuration (i.e. direct children of <cruisecontrol>.)

Project Changes

Once the machines have been configured, we then need to hook up the project to the machine. This can be configured like so:

<project name="distributedTest">
  <buildTarget xsi:type="singleTarget" name="buildAgent1" />
  <!– Other configuration goes here –>
</project>

This defines the build target as a single remote machine with the name of “buildAgent1”. When the project builds it will look in the list of machines and find the machine with the name of “buildAgent1”. This machine will then receive the request to build.

This can also be expanded to allow configuring the local build:

<project name="distributedTest">
  <buildTarget xsi:type="singleTarget" name="buildAgent1" runLocal="OnFailure" />
  <!– Other configuration goes here –>
</project>

At this point I am thinking of allowing three runLocal options:

• Never – never run a local build
• OnFailure – run the build locally if the remote build fails (e.g. the machine cannot be contacted or rejects the request)
• Always – always run a local build

The default would be OnFailure.

Handling Multiple Machines

Once we have the simple scenario, we can look at expanding this to handle multiple build targets. This would be done by adding a new build target type:

<project name="distributedTest">
  <buildTarget xsi:type="multipleTarget" runLocal="OnFailure" number="All">
    <targets>
      <singleTarget name="buildAgent1" />
      <singleTarget name="buildAgent2" />
    </targets>
  </buildTarget>
  <!– Other configuration goes here –>
</project>

This expands the original singleTarget by allowing multiple targets to be specified. The targets are specified with the same singleTarget definition (allowing re-use.) The runLocal attribute will work in the same way as runLocal on a singleTarget (note, runLocal will be ignored on the child targets.)

The number attribute specifies how many target machines to run on. This can be either a number or the keyword “All”. When “All” is specified the build will be run on all the specified targets.

Using Configured Machines

The above configuration handles most of the base scenarios, the only one uncovered is using configuration. This will be a new target type:

<project name="distributedTest">
  <buildTarget xsi:type="configurationTarget" runLocal="OnFailure">
    <configuration>
      <namedValue name="Browser" value="Firefox" />
    </configuration>
  </buildTarget>
  <!– Other configuration goes here –>
</project>

When the project runs it will try to match a machine with the same configuration (note there are no limits on what configuration values can be specified, so it is up to the administrator to decide what values are available.) This can then be combined with the multipleTarget to handle different configuration combinations:

<project name="distributedTest">
  <buildTarget xsi:type="multipleTarget" runLocal="OnFailure" number="1">
    <targets>
      <configurationTarget>
        <configuration>
          <namedValue name="Browser" value="Firefox" />
        </configuration>
      </configurationTarget>
      <configurationTarget>
        <configuration>
          <namedValue name="OS" value="Windows*" />
        </configuration>
      </configurationTarget>
    </targets>
  </buildTarget>
  <!– Other configuration goes here –>
</project>

Configuration Wrap-up

These are my basic plans for the configuration for distributed builds. These configuration elements cover all my basic scenarios, so in theory we can build a project in a number of different ways. Also this allows for people to add their own targets (round-robin, multiple-attempt, etc.)

However, there is still a side of the configuration that has not been covered – how to configure the remote machine? In my next post I will look into how I am thinking of doing that.

Stay tuned…

Distributed Builds: Part 3 (of 3+)

A Quick Recap

Previously (here and here) I started outlining my plan for adding distributed builds and some of the issues. I was planning on writing about how I think it can be configured, but I wanted to return to the design for a little bit.

Some Scenarios

The end goal is to try and make a flexible system that other people can expand if desired (similar to the underlying design for most of CruiseControl.NET.) To help with this, I’ve put together a few scenarios of how I see people using this functionality.

Scenario #1: Local Build

This is pretty much how CruiseControl.NET currently works. A build is triggered on the server and runs locally. I’ve included this scenario since we need to ensure that it still works!!

Scenario #2: Single Remote Build

In this scenario a project is mapped to one or more remote machines. The first machine that is available builds the project, otherwise the project will be built locally. This looks something like:

The remote server can be a list of servers, but it is only ever triggered on one remote or local server.

Scenario #3: Multiple Remote Builds (Pick x of y)

This scenario involves multiple remote servers all building at the same time. The project is mapped to y servers, and it expects at least x to build. This looks something like:

The project does not care which servers build the project, just that at least x servers build it.

Scenario #4: Multiple Remote Builds (All)

This is really an extension of the above scenario, but stating that all the remote servers must build the project. But it is slightly different from the above as the project needs to check that all the servers have built it.

Scenario #5: Multiple Remote plus Local

Again, another extension of scenario #3, the difference being the local server is included in the list of servers doing the build. This looks like:

There are two variations on this scenario, but they are reasonably minor. The first variation is the local server is included as an optional server (e.g. it is included as one of the x servers.) The second variation is the local server must always build (e.g. it is not counted in the x servers.)

Scenario #6: Multiple Builds (At least one of each configuration)

This scenario is a specialised case of #4 – the remote servers have different configurations (e.g. Windows, Linux, Mac, etc.). The build must run on at least one of each configuration. This looks something like:

The key here is that for each configuration the build must run at least once. There can be multiple machines with the same configuration (but there does not have to be.) The local machine can also be included as one configuration (optional.)

Extended Scenarios

While I see these as my initial scenarios, they can also be chained together. For example, a remote server could pass a request onto other remote servers:

This would people to build not only build farms, but build pipelines where builds are shipped out over a potentially large distribution, with (hopefully) a simplified set of configuration options.

Sidestep Finished

Anyway, that enough of my sidestep, in my next post I WILL start looking into the configuration.

Stay tuned…

Distributed Builds: Part 2 (of 2+)

A Quick Recap

In my last post (here) I started exploring distributed builds in CruiseControl.NET. In it I listed a number of issues to investigate, plus two scenarios that I’m planning on implementing. These scenarios are:

1. A server sends out a request to another server to build an entire project
2. A project sends out a request to a server to build part of itself

Both of these will use the same underlying communications model, they will just call it in slightly different ways.

The Plan

.NET already has a few communications options built into it, from simple sockets (a.k.a. roll-your own) through .NET Remoting right up to WCF. Previously we’ve used .NET Remoting, but this is a technological dead-end (just ask Microsoft!) It also has a number of issues, some of which have (and continue to) hurt us before.

So, what is the recommended framework – WCF (at least according to Microsoft). Previously this has not been an option for us as Mono has not supported it (yes, there was Project Olive, but this was not progressing when we last checked.) But times have changed, Silverlight has come and nudged Mono into the realms of WCF, so now Mono at least supports the Silverlight subset of functionality (plus there is talk of implementing most of the WCF stack.)

So with this in mind I’m not going to stress over communications, instead I’m going to use WCF (and yes we can review this decision later.) This helps in resolving some issues (secure channels, communications through firewalls, etc.)

Next, I’m going to treat the remote machine as a dumb server – it will not know anything about the caller. The caller will need to do all the work for the communications. While this is not necessarily the best solution, it is the simplest and allows me to test out some basics (like what to pass around.)

At the moment the remote machine will just be another CruiseControl.NET instance – this means all the logic for performing builds will already be there. The majority of the work will focus on the logic flow and the communications.

The Messages

There will be the following messages in the system (at least initially):

The calling server will hold the master copy of the configuration, while the remote server will have the individual build processes. Each server will have its own logs – the remote server will have just the logs for the builds, while the calling server will have its own log plus a merged copy of the remote log.

The initial message in the sequence is “Check build…”. This message is basically a request to the remote server asking can you build this project? It is up to the remote server to handle the logic of which builds are accepted.

If the first message is accepted, then the calling server then asks the remote server to perform a build (the “Build…” message). At the same time it will pass the configuration for what is needed to build. The remote server will accept this configuration, parse it and then build. It will return a unique build identifier to the calling server – this allows the calling server to check on the status of a build.

The calling server will then periodically poll the remote server for an update (via “Check status…”.) The remote server will return a status report containing all the necessary information (overall status, project item statuses, etc.) The calling server then uses this to update its own internal picture of what the remote build is doing.

If the “Check status…” returns that the remote build has finished then the calling server will request the log (via “Get log…”.) This log is then merged with the main build log.

The final message is more of a “what-if” message. If the build is cancelled on the calling server, then it needs someway to tell the remote server to cancel as well. This will not immediately cancel the build, instead it will call the logic on the remote server to cancel a build (in a similar way to abort build in the current system.) The calling server will continue to poll until the remote server says it has finished.

Hacking the Integration Loop

CruiseControl.NET already has an integration loop (with triggers, queues, events, etc.), so all I need to do is hack into this loop and add the remote integration stuff. This can be illustrated as follows:

The blue boxes are the already existing processes (much simplified). The purple boxes are the new changes to get distributed builds working, with the matching red boxes being the new changes on the remote server.

Basically the current logic will be unchanged, there will just be some additions. The current project loop will be used, but there will be an extra step to check if the build runs locally or not.If the build is not run remotely, then the process will be unchanged.

If the build is run remotely, it will pass off control to the remote server. This will be similar to running a project locally, except with some infrastructure around handling the communications between the two servers. The actual running of the project will be the same whether it is a local or remote project (the Run locally box), the only difference will be which machine it is running locally on.

To Be Continued…

This is the basics of how I’m going to hack into CruiseControl.NET to allow distributed builds. In my next post I’ll look into the configuration required, and then move on to the actual implementation.

Stay tuned…

Distributed Builds

An Objective

One of the nice features that CruiseControl.NET lacks is the ability to perform distributed builds. A distributed build is where some or all of a build is performed on another machine – thus opening the possible to have Continuous Integration “farms”.

Unfortunately there is a simple reason why CruiseControl.NET does not have distributed builds functionality – it is hard to implement and even harder to do some both securely and simply.

A single CI server is reasonably easy to secure – since it is up to the program to decide what inputs to accept (note, I’m not talking about the security features that were recently added to CruiseControl.NET, this is looking at a deeper level, how does the program function internally.) As soon as a system becomes distributed it needs a way for its components to communicate. This communications channel then becomes a possible attack site for hackers!

The second challenge – simplicity – also ties into security. It can require some complex settings to secure a system so it cannot be hacked – yet at the same time the system needs to be simple enough that an administration can configure the system without too much headaches (this is even more critical when it is an open source system without anyone producing professional documentation!)

Finally, to round things off, a distributed system is not just a single system running in multiple locations – there are a whole range of ancillary functions that are needed. Is there a single controller or is it a collaborative system? How do you log and monitor the various components? Are artefacts passed around the system or stored in a central location? How do the components get updated? And so on and so on!

But I’m a sucker for punishment, so I am going to give it a try. At this time I have no intention of resolving all the issues (since I’m only doing this for fun!) I’m looking at putting in the base ground-work for this functionality, with the hope that other people will also want to contribute. As such my work in this area will start off small and (hopefully) grow in size to handle all the challenges.

You have been warned…

Some Basics

There are different ways that we can implement distributed builds, unfortunately the way the code is structured within CruiseControl.NET limits some of these options. So to start with I’m only going to consider two scenarios:

1. A server sends out a request to another server to build an entire project
2. A project sends out a request to a server to build part of itself

Scenario 1 is a typically client/server association. There is a centralised server (potentially one of many) that hosts the standard integration cycle within CruiseControl.NET. This server would poll the triggers and handle the initial queuing, etc. The change will occur when the project is ready to build – first it will check if the project should (a) run on a remote machine and (b) there is a remote machine available. If both of these conditions are met, the build request will be passed to the remote machine. Otherwise the project will run locally.

Scenario 2 is the more interesting scenario. Here a project is triggered (either as currently or via a remote build – scenario 1). It will progress through its tasks as per normal, but there will be a new task task – runRemote (or something similar). When this task is hit, it will bundle up all the sub-tasks and send them to the remote server to run.

Both of these are simple in concept, but they do have a few issues to cover:

Auditing

Every build generates an audit (or build) log. This log then gets stored in the artefacts folder in an XML format. For a distributed build the log needs to be shipped back to the calling server and merged with the main log.

CruiseControl.NET also has a nasty issue where the logs also contain all the results from the various tasks. This means the logs can be massive – so the last thing we want to do is load them into memory (or at least no more than we need to!)

Artefacts

Generally the objective of a build is to produce some sort of artefact. Some of these artefacts are ignored (if you only want to ensure the code builds, you don’t care about the results), while other artefacts are needed later on in the build (code analysis might run on the binaries produced by compilation.) Some artefacts are merged into the logs, while others are separate (e.g. text/xml vs. binaries).

So we need some way to specify which artefacts are to be passed around and some way of doing it (again without overloading memory).

Configuration

There needs to be some way to pass the configuration around between the different components. Worst case scenario is the administrator configures each server individually, but I think that approach is far too error prone! Instead the calling server should pass the necessary configuration to the callee. This way the correct configuration will always be available.

We also need to consider how to configure both the server and the client applications so they know to talk to each other, preferably also they don’t screw each other up.

Security

How can we limit communications so only authorised components can communicate with each other? How do we prevent external parties from eavesdropping on the communications and intercepting/modifying them? Finally, how do we make the communications open enough to go through firewalls, etc. without rewriting an entire communications stack!

Yes it is going to be fun

That’s All… For Now!

This is a start on looking into this issue. I’ve more listed some of my issues and concerns, plus areas that need to be investigated. In my next post on this topic I’ll start to look at some proof of concept work I’ve done in this area.

Stay tuned…

Reducing Strings: Transferring Data

A Multi-OS Challenge

CruiseControl.NET has gone through a few iterations of file transfer. Prior to version 1.4, the only files that could be transferred was the build log and other special files like the statistics or RSS feed. To handle these files there were special methods in the Remote API, and all they did was transfer the entire file across the network as a single string (serialised via .NET Remoting).

With CruiseControl.NET 1.4 we added the ability to transfer any file that was located in the artefacts folder of a project. This was a new method in the Remote API, but it now allowed the remote client to ask for a file. If the file was there, it was transferred across the web. And this is where things got interesting!

Originally, this method generated a MarshalByRef-based object that would connect back to the server and transfer the data a block at a time. This ensured that only small blocks of data were transferred, thus reducing the memory usage on the server.

Unfortunately Windows 2008 did not like this approach! IPSec blocked the object from connecting back to the server, so no data was transferred (yes, it worked on every other OS we tried, just one that stopped it.) After several frustrating attempts to resolve the issue, we gave up and went to transferring the entire file as a single byte[] buffer over the network. Once again memory usage went up

So, for CruiseControl 2.0 we are back to the issue of how can we transfer large files across the network?

Let’s Chat

The problem with transferring a single large block of data across the network is it needs to be loaded into memory first, and then transferred across the network. Even .NET Remoting does not like transferring large blocks across the network, but it does it. Under the hood .NET Remoting takes the large block of data and breaks it into smaller blocks, which then get transferred across the network. This is what our original approach did, but IPSec did not like it!

So, how can we go back to the block-based approach and get past IPSec? The answer is to move from our nice chunky interface, were .NET Remoting handles the blocks, to a more chatty interface (yes, I know, normally not a good practise, but sometimes we don’t live in an ideal world!)

The current situation looks something like this:

With .NET Remoting handling the blocking within the red arrow. The new approach will look like this:

Yes, it will be a multi-step approach. The steps are:

1. Open the file on the server
2. Transfer each block of data
3. Close the file on the server

This approach requires tighter coupling between the client and the server (sigh). When the client opens the file, a file identifier if returned. This file identifier is what the client then passes to the server for all subsequent operations.

So, why a file identifier? The identifier is used to identify which instance of the opened file is being used (the file is opened in read-mode so it can be opened multiple times). The server then maps the identifier to the stream instance (which is only held on the server). When each block is transferred, the stream is repositioned so the next block is ready. This means the client does not care where it is in the process, it only cares that there is more data to fetch.

The other downside for this approach is the file now needs to be closed on the server – other the number of open streams will slowly increase (i.e. a memory leak!) So closing the file will clean up after the transfer has been completed.

The other approach I looked at was for the client to pass in the starting position. But in the end I decided against this approach for two reasons:

1. It would require the client know what data it wanted (i.e. the position within the file). If it requested a position beyond the file length, then the server would need some error handling to not fail.
2. It would require opening the file, positioning to the correct location and closing the file every data fetch! I’m not sure on the performance of this, but it sounds slow.

Of course, if we run into issues with my current approach we can switch to the alternate and test it out, but for now, it’s the approach we are using

Being Helpful

Now, rather than forcing the clients to implement their own versions of file transfer (and potentially duplicate their code), I’ve added a new method to CruiseServerClient in the Remote API. This method is called TransferFile() and it literally does that – it handles all the logic of transferring a file from the server into a local stream.

To use this method is as simple as the following code:

client.TransferFile(projectName, fileName, outputStream);

Where client is an instance of CruiseServerClient (this can be generated from a CruiseServerClientFactory).

I have gone through and removed the old file transfer mechanism (in both the web dashboard and CCTray) and changed to using this method. This means all file transfers will now use the new approach and use less memory. However in a future post I’ll do some tests to see how much (if any) memory is being saved.

Coming Up

Now that we have a file transfer mechanism, the next step is to look at handling the new build log format. In my next post (or posts) I’ll look at the modifications to retrieve the build data and the changes to the dashboard.

Stay tuned…

Reducing Strings: A Quick Look Back

Where Have We Come From?

Before I start modifying the remote interfaces to use the new stream-based data storage, I thought I’d take a quick look back at the memory performance of these interfaces. Again I’ll be using ANTS Memory Profiler 5 to look at how memory is being used.

Just to show the impact of strings in the system I’ll take a look at the original system before I added caching, and then after to see what impact this change had. These will then provide the baselines for looking at how the stream changes can reduce the memory footprint of the CruiseControl.NET server.

But before I get into the analysis, let’s revise how data is sent from the server to a client (e.g. CCTray, the dashboard, etc.) The entire build log is stored in a single file on disk. The client makes a request to the server via .NET Remoting. This then opens the file, reads it all into memory and returns the entire file back to the client as a string. The caching changes modify this procedure only by ensuring there is a single instance of each build log in memory – subsequent calls to retrieve the same log will use the cached log (note, this is cached in memory).

The Method

This is not as nice and tidy as testing the memory usage for a project. The server needs to be up and running and the client then makes the call to the server. However since we are not interested in the client memory usage (yet), we need to be monitoring the server and we cannot directly tell when a memory usage scenario is beginning. To get around this limitation I wrote a small application (stress client) that makes multiple calls to the server to fetch the latest build log for a project. The build log I am fetching is 6Mb in size, the application will attempt to download this log concurrently multiple times to give the profiler plenty of attempts to analyse. On my memory I was able to get four concurrent instances connecting at once, each instance fetched the same build log ten times.

With this in place, I started the profiler and started profiling the server (ccnet.exe). This was run without any start-up parameters and left running for the duration at the test. I waited 30 seconds after starting the application and after the test had finished to make sure the relevant data was collected. After the 30 seconds for start-up had passed I ran the stress client and waited for it to finished. 30 seconds after the client had finished I shut down the server. I performed two sets of tests. The first set of tests I just ran the profiler and collected the memory stats. The second set of tests I captured the memory usage every 5-10 seconds during the test (this was done manually).

These tests were run three times, with the following versions:

1. This is the original CruiseControl.NET server before the caching was added
2. This is CruiseControl.NET server with caching added
3. This is CruiseControl.NET server with caching added, plus an extra modification to garbage collect after each build log fetch

The 1.5.0 codebase was used for all three versions. Only the get build log method was modified for each version.

The Data

These are the results from the original version (#1):

For the original version, just accessing the build log allocated 234Mb of memory to the server – of which most is used by the large object heap.

Changing to the modified version of the server we get the following results:

The first thing to notice is the private bytes is also half the original version – and the large object heap is very flat. In the original version a new string was added to the heap every time a client accessed the build log, in the new version there is only one version – hence the reason for the heap being much flatter. However the application is still using a lot of memory – there is a total of 80Mb assigned to all the heaps.

So, the final change is to force garbage collection:

Again, there is a drastic reduction in memory – this time there is only a maximum of 40MB in all heaps – and the total memory allocated has also been reduced. However, looking at the large object heap, there has been no change – so the question is what has changed and why?

Digging Deeper

The ANTS memory profiler has the ability to take a snapshot of what is currently being used by an application. This shows what objects have been allocated and what is associated with what. The only down side is the snapshots have to be manually taken, but the graphs help to show where snapshots need to be taken. These snapshots can then be viewed in the application to see where the memory is being used and what is holding onto the memory.

For this analysis I only took snapshots of versions 2 and 3 to see exactly what was using memory. Version 2 showed the following information:

While version 3 showed the following:

This shows that string and byte[] has the most memory usage in both versions, but it is the amount of byte[] memory that has changed. The following shows why this is the case:

The remoting infrastructure is generating blocks of byte[] data (byte[4096] to be exact) which are then being transmitted to the client. When the data is returned to the client the remoting infrastructure is breaking it down into blocks of byte[] data, these are what then get sent over the wire. These blocks are only 4096b in size, so they are not stored on the large object heap, but the large number of them (4,800 in the example for version 2) means they are taking up a large amount of space. So, in order to reduce memory usage we also need to look into this area also. This is important to note, because in CruiseControl.NET 1.5.0 (and earlier versions) all data transfer is handled by sending a single block of data. We did try during the development of 1.5.0 to do a block transfer, but IPSec in Win2K8 blocked it, so we had to reverse the changes. So we need to find a permanent solution to this issue that will work on all OS versions!

In Summary

We had a quick look at memory consumption under 1.5.0 seeing what could be done to reduce the amount of memory usage. However we’ve pretty much run into a stumbling block with the current log size and that it is a single (huge) block of data. For CruiseControl.NET 2.0 we have already split the build log into smaller files, but we also need to look at what can be done to reduce memory usage while transferring over the wire.

So, in my next post I’ll take a look at an alternate transfer mechanism. Stay tuned…

Reducing Strings: Converting Tasks

A Quick Review: Task Classifications

A few posts ago (here), I classified the tasks into how they generate output. Broadly, the output can be split between the build log or directly generated to the file system. There is also a third type of output – external output. This is output that is generated by an external application and stored directly to the file system.

These three types all need to be modified to generate index entries, so the results can be accessed later on. Tasks that write to the build log (e.g. via strings in the old version) or those that write to streams (e.g. direct output to the file system) are easy enough – we already have methods on TaskContext to  generate indexed streams. However external applications that generate files directly need consideration.

So, for this post I am going to convert two tasks, one that generates build log data and the other that manipulates external data. The tasks I will convert are:

• ExecutableTask
• NDependTask

This is also the order of complexity of the conversion I omitted a task from the internal generation, as those tasks often have additional (and complex) logic. I will hopefully look at them in a future post.

General Execution: ExecutableTask

One of the most basic tasks in the system is the executable task. This task is used for executing any external application, which also makes it one of the most versatile tasks. This task attempts to execute the application, nothing more, nothing less. Any output to standard out or standard error is captured by CruiseControl.NET and written to the build log.

In the current version of CruiseControl.NET (1.5 or earlier), generating the build output from the executable task is a two stage process. First, everything is written to a StringBuilder and then dumped into a string. In the second stage, this string is converted into XML.

For my first version of the conversion to streams, I did a similar process – except using streams. The process executor would write the standard out and standard error to a stream. Once this had finished, it would then go through and convert the results into XML. This is what I used to check the proof of concept was working.

However, this was still a two stage process – generate the results and format them. There is no reason why this is required – it should be possible to generate the results directly as XML – which is what I have done.

Most of the changes to handle this were done in ProcessExecutor. Previously I had modified the signature for Execute() to take in two streams – one for standard out and the other for standard error. These streams were then wrapped in StreamWriter instances and passed onto the runnable process. When the external application wrote to standard out or standard error, the output was sent to the corresponding StreamWriter, which sent the data to the stream.

My new change was to introduce a new class – XmlStreamWriter. This class wrapped a StreamWriter and overrode some of its methods. The most important method it overrides is WriteLine() – when this method is called it wraps the line in an XML block and ensures the data is valid (e.g. has no invalid XML characters). It also appends a line number and the date/time, just to make the output more meaningful. The other methods that I overrode were to start/end the XML document (since an XML document can have only one root) and to clean up when necessary.

Now, all ExecutableTask does is start two result streams and pass these through the levels to ProcessExecutor. It also tells the logic to wrap these streams in the new XmlStreamWriter class, which does the automatic conversion. And that’s it – nice and easy. the infrastructure does the rest of the work – generating the index entries, writing them to the log file, ensuring the output is XML, etc.

Analysing Code: NDependTask

The NDependTask calls an external application called NDepend. This application performs a static analysis on a codebase and produces a number of statistics about the code, including complexity, cyclic redundancy and interrelationships. The reason I choose this task is it generates a number of files which need to be included in the build results. These files include both XML data and non-XML data (e.g. images). Additionally, since it is an external task it generates standard out and standard error to be included.

The first part of the conversion is easy – this is almost identical to ExecutableTask – initialise the streams for standard out and standard error, pass them through the layers to where the application is actually executed and store the result for later.

It’s the second part that requires some additional changes. After the external application has finished, the task checks for any new files that were generated. These files are then imported into the results.

To handle this I added a new method to TaskContext – ImportResultFile(). This method has a similar method signature to CreateResultStream() but with two important differences – it has a filename argument and it does  not return anything. This method simply generates a new index entry and moves the file to the results folder. Now external files can be included in the results index.

A Simple Path

Now we have a nice simple migration path for tasks to use the new index. For tasks that generate data they can use the CreateResultStream() method on TaskContext. This will generate a new stream for the data, plus the associated index entries. These streams can then be passed into ProcessExecutor for handling the stdout and stderr output.

For tasks that generate files via an external application, the ImportResultFile() method can be used. This moves the external file into the data folder, plus generates the associated index entries. This also offers the choice to move or copy the files so the calling task can tidy up if necessary.

These two methods, either separate or together, handle most of the data generation for the current tasks/publishers. There are a few tasks/publishers that won’t like this approach, mainly because they need to manipulate files (i.e. read/write access), but as these tasks/publisher tend to be outside the norm I’m going to ignore them for the moment.

The next challenge is to look into accessing these new index entries and the associated files. So stay tuned for a review of how we currently use the build logs and how it will change…

Reducing Strings: Additional Task Details

An Extra Feature

In looking at reducing strings (and thus memory usage) I realise we also have another opportunity to enhance CruiseControl.NET. A while back CruiseControl.NET was enhanced to display current task status. This would display a list of all the tasks for a build, and the current progress through the tasks. However, this information is only ever stored in memory – as soon as the server restarts or another build is triggered (even if it is just a source control check) the information is lost!

With the new changes it is easily possible to persist this information – plus we can go one better and associate the task output with the task information So in this post, I’ll look into how these changes work.

Currently

When the task outputs are added to the results they are added to an array (this is all done in the task context). These are accumulated per context – and when a context is completed the outputs are merged into the parent context. When the build is completed, all the outputs are written directly to the build log, one element per output.

Each output has a number of attributes – task type, result name, task identifier and data type. Some of these are used to identify the output, while other attributes identify the task that it is associated with.

The Changes

The changes fall into two basic categories: first the task context now has the task level attributes (task type and identifier) and the outputs are now associated with a task element in the build log. To handle this, I have split TaskResultDetails into two classes: TaskResult and TaskOutput. TaskResult holds the task level attributes, plus the outcome of the task, while TaskOutput holds the output details.

The task context has been expanded to allow the task type and identifier (currently the task name/description) to be set. At the same time, a new TaskResult is generated for the context (remembering that a context is being generated per task). To enable linking between the task status and the outcome to be stored, the context now requires an IIntegrationResult instance. The status will be set here (it is currently) and the context will pull the status when the context is finalised.

The ImportResultFile() and CreateResultStream() methods have been modified to generate TaskOutput instances now. These instances get stored in the associated TaskResult for the context.

Finally, TaskResult now contains a list of child TaskResult instances. When a context is merged into a parent context the associated TaskResult is appended to the child instances of the parent. When the project-level context is finalised, the associated TaskResult is what is serialised.

Serialisation

Both TaskResult and TaskOutput have a WriteTo() method that takes in an XmlWriter. When these methods are called, they will output the properties of each instance to the writer, and then call the WriteTo() method on any child items (either TaskResult or TaskOutput). To ensure this is written to the build log, XmlIntegrationResultWriter is responsible for calling this method on the project-level context.

The following XML is generated:

<build date="2009-11-12 03:37:37" buildtime="00:00:00" buildcondition="ForceBuild">
  <result type="project" identifier="Test" outcome="Unknown">
    <result type="exec" identifier="exec #1" outcome="CompletedSuccess">
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\233e1e69d-f0ab-441c-bc6f-030e52b61f58.data" name="stdout" data="data/xml" />
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\23\2d1a5ede-b2c8-432e-bf87-af01a75ebd2d.data" name="stderr" data="data/xml" />
    </result>
    <result type="exec" identifier="exec #2" outcome="CompletedSuccess">
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\23\7f9f9bfe-2d5d-43b1-be1f-a61303701b72.data" name="stdout" data="data/xml" />
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\23\1b65cf49-4902-441f-848b-c2f22234d6b7.data" name="stderr" data="data/xml" />
    </result>
  </result>
</build>

Now it is easy to see which tasks ran, what the outcome of the tasks were and what output they generated.

 

In Conclusion

This (hopefully) concludes the changes to the build engine on the server for the stream-based output. In the next set of posts I will delve into using the new output.

Stay tuned…

Reducing Strings: XmlLogPublisher is Dead, Long Live XmlLogPublisher

Good Bye XmlLogPublisher

As part of the changes for implementing the task context I am removing XmlLogPublisher. This is to provide a standardised way of generating the log – while still exposing the required functionality to the other tasks that need it.

In the current version of CruiseControl.NET users need to add an xmllogger element to their project configuration (preferably in the publishers section, but there is no reason why they can’t put it in the tasks section). When this task is run it will generate the log file, which can then be used by sequent tasks. However, to complicate things, some tasks generate their own version of the log file (normally in memory) – e.g. StatisticsPublisher and EmailPublisher. Plus if the user adds a publishers section and forgets to add the xmllogger element, then no log will be written!

Note: if the user does not define a publishers section, a publishers section will be generated by default that contains the xmllogger element.

So, to standardise and simplify things, I am moving the functionality from XmlLogPublisher into TaskContext. This functionality will be called automatically by Finialise() in TaskContext (this method is called when an integration finishes).

Currently XmlLogPublisher generates the folder for saving the log in, generates the name of the log file and then writes out the log (via XmlIntegrationResultWriter). I have moved these into TaskContext as separate methods (as GenerateLogFolder(), GenerateLogFilename() and WriteCurrentLog()). The Finalise() method calls all of these, but I have exposed them as public methods so they can be called from other locations.

Additionally, I made a small change to XmlIntegrationResultWriter. Because I want to write the current result entries out via the writer, I changed the constructor to take in an enumeration of TaskResultDetails. I also changed the WriteTaskResults() to output these results (previously it was outputting the data from the tasks). This required changing the statistics and e-mail publishers to use TaskContext instead of XmlIntegrationResultWriter. Unfortunately it also means these are both breaking at the moment, as they are expecting a valid log file to be generated, and it has now been changed to only have index entries! In a future post I will look at modifying these so they work again

Finally to round out this change, I added another new method to TaskContext. This method (GenerateResultsSnapshot()) will generate the current snapshot of all the results. This breaks the normal working of TaskContext, in that it allows a child context to access a property on the parent context. This means that property (the list of index entries) now needs to be locked before it can be accessed. I did consider using the reader/writer lock instead of the general monitor lock, but it sounds like the .NET 2.0 version is flawed, so I decided to avoid it.

So, in saying farewell to XmlLogPublisher, I have moved most of its functionality to TaskContext. I’ve also expanded both TaskContext and XmlIntegrationResultWriter to handle writing the index entries to the log file.

Hello XmlLogPublisher

Having removed most of the functionality from XmlLogPublisher, I rebuilt it to use the new functionality from TaskContext. Why? So it is still possible to generate a build log part way through the build process. This is required for those tasks/publishers that want to do something with a physical build log file. For example, the FTP publisher might want to send the build log to a remote machine, the package publisher might want to include it in a package, etc. The new version allows this functionality.

However, I also expanded some other functionality to make this even easier. As part of the changes in removing the functionality, I removed the log directory configuration setting to Project. While the new XmlLogPublisher still has this setting, I wanted to expose the project-level setting to the publisher. Currently tasks/publishers work in isolation – they don’t know anything about the project that owns them (beyond what is passed about in the IIntegrationResult.) So, I wanted to expose project configuration settings, but in an immutable way.

To do this, I have added a new class called ProjectConfiguration. This class exposes a number of the projects from Project in an immutable form. These are loaded from the project when the class is initialised and are read-only. I then added this new class as a property on TaskContext. Now, any task with an associated TaskContext can access these project configuration settings. At the moment there are only a few settings, but over time I will expand ProjectConfiguration to include most of the configuration settings (yes, it is a work in progress!)

So, now the new XmlLogPublisher can use its own log folder, the project’s log folder (if set) or the project’s artefact folder. At the same time, other tasks can now access project settings in a safe, immutable way.

So, that’s the first set of changes – XmlLogPublisher has been changed to fit the new structure, and the index entries are now being written to the disk. In the next set of changes I’ll look at modifying some tasks/publishers to use the new indexing methods.

Stay tuned…

Reducing Streams: Stream Metadata and Indexing

Data about Data (and so it goes…)

In my last post (here) I reviewed all the tasks and publishers currently implemented. As part of the review there were a number of tasks/publishers identified that generated output that is not currently included in the build log. As part of the changes, we want to be able to handle these files with a standard process as well.

In general, the strategy is to let each task write directly to disk, instead of writing to memory (in a string). It is the task’s job to decide what to write and how. In an even earlier post (here) I introduced the concept of a TaskContext. This context is responsible for starting new streams and storing references to them. While this concept is still valid, it needs a couple of extensions.

My Initial Plan

In my initial plan I was going to let the task choose the file name. The task would call an open stream method on the context and tell the context the file name and a result type name. The context would then open a new file stream using the file name in a folder for the build. If the file name was already in use, the context would generate a unique name (normally by inserting a literal and number into the name). To handle multi-threading, each context required its own folder, and when a context was finished, the owner would then move all the files from that folder into its own folder, again ensuring uniqueness of the names.

This added some extra complexity, just to produce easily readable file names. It did not guarantee that a file name would match a the expected task (i.e. there could be a nunit.xml that did not come from the NUnit task). The task type name was only stored in the index file, and the index file name was not unique (it would overwrite previous versions!)

Additionally, the context only handled generating new streams, it didn’t handle importing files generated externally (e.g. output from external applications). These would still exist outside the generation process, and they were not included in the index file.

Finally, to handle the duo-stream output from external tasks, I had put a stream merging process into place. This would literally take two (or more) streams and merge them together, plus it also converted the streams into an XML format (e.g. one element per line, with a line type per element). Nice maybe, but of questionable value since it adds additionally processing overhead!

Changing the Process

In retrospect I made things harder by trying to be nice and also by having the entire indexing outside of the current way of doing things (since this would also have a minimal impact on the current way of doing things.) Now, I think that was a bad decision – these changes are going to be breaking anyway, so lets try and do things properly!

First, unique file names: the name of the file needs to be unique, no matter which task generates it, whether there are duplicates of a task type, or multiple files of the same name being merged. The uniqueness needs to be enforced even when different threads generate files, and whether the files are generated internally or externally. Rather than inventing a new process for unique names I’ll use a tried and true method – GUIDs. When a new result stream is opened the physical name of the stream will be a GUID – no more, no less.

The index entries for each stream will contain the necessary data to identify what each file is for and the type of data contained. These metadata items are:

• Task type: the type of task that generated the result. This will be the name of the task in the configuration (e.g. nunit, msbuild, nant, etc.) and it will be up to the calling task to pass in.
• Result name: this is the internal name of the file. Examples would be stdout, stderr, test results, etc. Again, it is up to the task to pass in this name and it should be unique per task type (although not sure how to enforce this!)
• Task identifier: this is an identifier for the actual instance of the task. Again, this will come from the calling task and could be something like the user-configured task name, an internally generated identifier or the date/time. Basically, it is used to group streams together within an actual instance of a task.
• Task order: this is the order in which the result was generated. The context will generate this number and associate it with a result. When a child context is merged, the owning context is responsible for ensuring the order numbers are unique. As such this identified the order in which tasks are completed.
• Data type: this is the type of data the file contains. Ideally, this will be the mime type of the data, so it can be used in serving the files via the dashboard. Again, this will need to come from the calling task.

Creating Index Entries

Now that I know what I want to store, and who is responsible for populating streams, the question is how are these index entries and streams generated?

Based on the sources of data, there needs to be two ways of generating the indexes. First, there are the internally generated entries. In this scenario the task will request a stream and pass in the required metadata. The context will generate the stream and at the same time generate an index entry. After this, it is the responsibility of the task to manage the stream (including closing it!)

The second scenario is an externally generated file. In this case the task will tell the context where the file is, plus the required metadata. The context will then import (move/copy?) the file into the build folder and add the index entry. The task does not need to do anything else.

And Finally, Some Context

I’ve written a bit about the context and some of how it works, but I still haven’t covered how it is generated or how it writes the indexes.

An initial context will be generated when a project starts building. At the moment this will be associated with the Project, although later on I may move it (especially when we come to splitting the context from the configuration). As each task is run, a child context is started for the task. The task then runs within this context, generating its own indexes. If the task starts child tasks (e.g. parallel task, sequential task, etc.) it must generate a child context for each child task.

After a task is completed, the associated context is finalised (either by the project or the parent task). Now an important point – each task or project only works with one context, and no other task or project will change that context! This ensures that the context will not be messed up by two tasks trying to modify the context at the same time. There is only one exception to this rule – when a context is finalised. At this time the context is locked so the owning task cannot use it anymore and all the index entries are transferred to the parent context.

If the context does not have a parent context (i.e. it is the project-level context), it will generate the final index. This index will then be included in the standard build log. This will require a slight shift in the way the build log is generated. Currently the build log is generated by the xmllogger task, which means it can be generated at different points in a build. This task will be removed from the available tasks, and the build log generated automatically at the end of the process.

At this point you are wondering how can publishers access the build log? For example, the e-mail publisher may want to generate an e-mail containing the results of the build, the FTP publisher might want to send the log to a remote machine, etc. This will be handled in two ways:

1. Each task context will have access to the current list of indexes. This will need to be accessed in a thread safe way, just in case the user has configured a multi-thread project. Either way, the task context will have access to the entire set of index entries from its parent and all ancestors, all the way to the project context. This means the task will only be able to access index entries from finalised task contexts.
2. At will be possible to generate a build log at any point. These build logs will only be in-memory, but will contain all the current build details, plus the current list of index entries (see point 1. above). At the end of a build this will be the same file that is generated for the overall build log. The new build log structure will replace the build results that currently exist with the index entries.

Time To Work

This describes my plan on how I will implement things. Coming up next, some actual implementation details.