In my last post (read it here) I showed the results of a new documentation process for CruiseControl.NET. The aim was to move the documentation from being split over two different locations (code and the wiki) into one (the code) and to make it easier for developers to write documentation (plus it also standardises the documentation!)

To do this I have used doc comments in the code (yes it bloats the code, but with Visual Studio these comments can be collapsed). Most of the time I tried to use the standard documentation tags (e.g. summary, remarks, example, etc.), but unfortunately the standard tags don’t cover all we need

The good news is when Microsoft wrote the doc comment extraction application they didn’t enforce the tag types. This means we can add additional tags and they will happily be extracted. Of course, we don’t want to go overboard as the intellisense in Visual Studio doesn’t know about them (does anyone know of how to add new doc comment tags to Visual Studio?)

The new tags I have added for classes are:

The new tags I have added for fields and properties are:

Additionally I expanded the code tag (this already exists) to have a title attribute. This will apply the title to the code example block – Confluence allows code titles, so it nicely ties the two together.

The documentation generator mainly uses reflection for the documentation. However this generates the bare bones of each article. For example, this is what the state element looks like:

This is because this class has no doc comments (none at all!) associated with either the class or the fields/properties that are reflected.

Adding the summary tag adds the description:

Adding a title tag:

And a version tag (this was a guess):

Finally, to add an example I used the example tag (with the new title attribute):

Finally the configuration element data is added by adding the default and version tags to the actual property definition:

Notice how this was a progressive enhancement of the documentation. If we added a remarks tag to the class documentation we would get a notes section (this element doesn’t have any notes at the moment).

The doc comments for the class look like this:

/// <summary>
/// The File State Manager is a State Manager that saves the state for one project to a file. The
/// filename should be stored in either the working directory for the project or in the explicitly
/// specified directory. The filename will match the project name, but will have the extension .state.
/// </summary>
/// <title>File State Manager</title>
/// <version>1.0</version>
/// <example>
/// <code title="Minimalist example">
/// &lt;state type="state" /&gt;
/// </code>
/// <code title="Full example">
/// &lt;state type="state" directory="C:\CCNetState" /&gt;
/// </code>
/// </example>
[ReflectorType("state")]
public class FileStateManager : IStateManager

And for the property:

/// <summary>
/// The directory to save the state file to.
/// </summary>
/// <version>1.0</version>
/// <default>The directory CCNet was launched from.</default>
[ReflectorProperty("directory", Required=false)]
public string StateFileDirectory

Where possible the document generator pulls the data from the attributes and property definition. For everything it tries to find the relevant doc comments – leaving them empty/blank if not found.

What do you think so far?

I’ll do one more post – how the document generator works from an external viewpoint, and how to get the documentation into the wiki.

Yes, it didn’t take me very long

In my last post (here) I said I was going to go through all the documentation and ensure that it is:

1. Up to date
2. Valid
3. Standardised

For the first step in this task I generated a list of all the types in the Remote, Core and WebDashboard projects that had the ReflectionType attribute (this means a type can be used in the configuration).

This identified 176 classes that needed to be checked. Now not all of these classes are used as tasks, triggers, source control blocks, plug-ins, etc., some are ancillary types to provide sub-information (e.g. named values, user details, etc.)

This data was dumped into an Excel spreadsheet, so I could go through each and every class to check the documentation. As you might have guessed, this didn’t last very long…

The first class I checked was DefaultQueueConfiguration – which maps to the queue tag. Now, as classes go, this one is fairly simple – only three properties that are reflected. But, after half an hour of flicking backwards and forwards between the documentation and the code I was thinking why do we have the data split over two places?

Instead I copied all of the documentation from the wiki into the code as doc comments (including some reformatting). This means everything is in one place now – yay!!! However, this doesn’t answer the problem of documentation – it just means duplication of data

So, rather than do more documentation I quickly put together a little application that used the XML comments and reflection to automatically generate the documentation. Now the code is back to being the single source of truth, plus the documentation will now be more consistent (since it is automatically generated).

Here are some before and after shoots of how the documentation looks (they are big, so click on the thumbnail to see the full version):

Now I am back to the process of reviewing documentation – in a future post I explain how the process works and what is required.

As part of the preparation for the 1.5 release (hopefully soon) I thought I’d go through and review the documentation to ensure it is:

1. Up to date
2. Valid
3. Standardised

All the documentation for CruiseControl.NET is stored at http://confluence.public.thoughtworks.org/display/CCNET. This is an online wiki with its own formatting mark-up (it uses Confluence).

Now, the question arises of how to review the documentation?

For CruiseControl.NET the source of truth is the code, so I need to go through both the documentation and the code to ensure everything matches. What would be really nice is if everything was documented in the code – but I won’t comment on what our documentation comments are like.

So, that leaves a nice challenge of checking everything manually.

Basically, I’m planning on doing it two ways. First, I will go through the code and verify all the items in the documentation. Hopefully this will cover most of the pages in the wiki. Once I’ve done that I will then go through the remaining pages and check them.

Now for the bad news, I’ve extracted all the classes in CruiseControl.NET (from the Core, Remote and WebDashboard assemblies) and there are 176 classes that can be reflected – which means 176 classes that can be included in the documentation.

So, this may take some time…

Notes: As a side task I am also planning on importing the documentation from the wiki to the code. This means in future we should be able to automatically verify some of the documentation (hopefully!)

Where Have We Come From?

Before I start modifying the remote interfaces to use the new stream-based data storage, I thought I’d take a quick look back at the memory performance of these interfaces. Again I’ll be using ANTS Memory Profiler 5 to look at how memory is being used.

Just to show the impact of strings in the system I’ll take a look at the original system before I added caching, and then after to see what impact this change had. These will then provide the baselines for looking at how the stream changes can reduce the memory footprint of the CruiseControl.NET server.

But before I get into the analysis, let’s revise how data is sent from the server to a client (e.g. CCTray, the dashboard, etc.) The entire build log is stored in a single file on disk. The client makes a request to the server via .NET Remoting. This then opens the file, reads it all into memory and returns the entire file back to the client as a string. The caching changes modify this procedure only by ensuring there is a single instance of each build log in memory – subsequent calls to retrieve the same log will use the cached log (note, this is cached in memory).

The Method

This is not as nice and tidy as testing the memory usage for a project. The server needs to be up and running and the client then makes the call to the server. However since we are not interested in the client memory usage (yet), we need to be monitoring the server and we cannot directly tell when a memory usage scenario is beginning. To get around this limitation I wrote a small application (stress client) that makes multiple calls to the server to fetch the latest build log for a project. The build log I am fetching is 6Mb in size, the application will attempt to download this log concurrently multiple times to give the profiler plenty of attempts to analyse. On my memory I was able to get four concurrent instances connecting at once, each instance fetched the same build log ten times.

With this in place, I started the profiler and started profiling the server (ccnet.exe). This was run without any start-up parameters and left running for the duration at the test. I waited 30 seconds after starting the application and after the test had finished to make sure the relevant data was collected. After the 30 seconds for start-up had passed I ran the stress client and waited for it to finished. 30 seconds after the client had finished I shut down the server. I performed two sets of tests. The first set of tests I just ran the profiler and collected the memory stats. The second set of tests I captured the memory usage every 5-10 seconds during the test (this was done manually).

These tests were run three times, with the following versions:

1. This is the original CruiseControl.NET server before the caching was added
2. This is CruiseControl.NET server with caching added
3. This is CruiseControl.NET server with caching added, plus an extra modification to garbage collect after each build log fetch

The 1.5.0 codebase was used for all three versions. Only the get build log method was modified for each version.

The Data

These are the results from the original version (#1):

For the original version, just accessing the build log allocated 234Mb of memory to the server – of which most is used by the large object heap.

Changing to the modified version of the server we get the following results:

The first thing to notice is the private bytes is also half the original version – and the large object heap is very flat. In the original version a new string was added to the heap every time a client accessed the build log, in the new version there is only one version – hence the reason for the heap being much flatter. However the application is still using a lot of memory – there is a total of 80Mb assigned to all the heaps.

So, the final change is to force garbage collection:

Again, there is a drastic reduction in memory – this time there is only a maximum of 40MB in all heaps – and the total memory allocated has also been reduced. However, looking at the large object heap, there has been no change – so the question is what has changed and why?

Digging Deeper

The ANTS memory profiler has the ability to take a snapshot of what is currently being used by an application. This shows what objects have been allocated and what is associated with what. The only down side is the snapshots have to be manually taken, but the graphs help to show where snapshots need to be taken. These snapshots can then be viewed in the application to see where the memory is being used and what is holding onto the memory.

For this analysis I only took snapshots of versions 2 and 3 to see exactly what was using memory. Version 2 showed the following information:

While version 3 showed the following:

This shows that string and byte[] has the most memory usage in both versions, but it is the amount of byte[] memory that has changed. The following shows why this is the case:

The remoting infrastructure is generating blocks of byte[] data (byte[4096] to be exact) which are then being transmitted to the client. When the data is returned to the client the remoting infrastructure is breaking it down into blocks of byte[] data, these are what then get sent over the wire. These blocks are only 4096b in size, so they are not stored on the large object heap, but the large number of them (4,800 in the example for version 2) means they are taking up a large amount of space. So, in order to reduce memory usage we also need to look into this area also. This is important to note, because in CruiseControl.NET 1.5.0 (and earlier versions) all data transfer is handled by sending a single block of data. We did try during the development of 1.5.0 to do a block transfer, but IPSec in Win2K8 blocked it, so we had to reverse the changes. So we need to find a permanent solution to this issue that will work on all OS versions!

In Summary

We had a quick look at memory consumption under 1.5.0 seeing what could be done to reduce the amount of memory usage. However we’ve pretty much run into a stumbling block with the current log size and that it is a single (huge) block of data. For CruiseControl.NET 2.0 we have already split the build log into smaller files, but we also need to look at what can be done to reduce memory usage while transferring over the wire.

So, in my next post I’ll take a look at an alternate transfer mechanism. Stay tuned…

An Extra Feature

In looking at reducing strings (and thus memory usage) I realise we also have another opportunity to enhance CruiseControl.NET. A while back CruiseControl.NET was enhanced to display current task status. This would display a list of all the tasks for a build, and the current progress through the tasks. However, this information is only ever stored in memory – as soon as the server restarts or another build is triggered (even if it is just a source control check) the information is lost!

With the new changes it is easily possible to persist this information – plus we can go one better and associate the task output with the task information So in this post, I’ll look into how these changes work.

Currently

When the task outputs are added to the results they are added to an array (this is all done in the task context). These are accumulated per context – and when a context is completed the outputs are merged into the parent context. When the build is completed, all the outputs are written directly to the build log, one element per output.

Each output has a number of attributes – task type, result name, task identifier and data type. Some of these are used to identify the output, while other attributes identify the task that it is associated with.

The Changes

The changes fall into two basic categories: first the task context now has the task level attributes (task type and identifier) and the outputs are now associated with a task element in the build log. To handle this, I have split TaskResultDetails into two classes: TaskResult and TaskOutput. TaskResult holds the task level attributes, plus the outcome of the task, while TaskOutput holds the output details.

The task context has been expanded to allow the task type and identifier (currently the task name/description) to be set. At the same time, a new TaskResult is generated for the context (remembering that a context is being generated per task). To enable linking between the task status and the outcome to be stored, the context now requires an IIntegrationResult instance. The status will be set here (it is currently) and the context will pull the status when the context is finalised.

The ImportResultFile() and CreateResultStream() methods have been modified to generate TaskOutput instances now. These instances get stored in the associated TaskResult for the context.

Finally, TaskResult now contains a list of child TaskResult instances. When a context is merged into a parent context the associated TaskResult is appended to the child instances of the parent. When the project-level context is finalised, the associated TaskResult is what is serialised.

Serialisation

Both TaskResult and TaskOutput have a WriteTo() method that takes in an XmlWriter. When these methods are called, they will output the properties of each instance to the writer, and then call the WriteTo() method on any child items (either TaskResult or TaskOutput). To ensure this is written to the build log, XmlIntegrationResultWriter is responsible for calling this method on the project-level context.

The following XML is generated:

<build date="2009-11-12 03:37:37" buildtime="00:00:00" buildcondition="ForceBuild">
  <result type="project" identifier="Test" outcome="Unknown">
    <result type="exec" identifier="exec #1" outcome="CompletedSuccess">
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\233e1e69d-f0ab-441c-bc6f-030e52b61f58.data" name="stdout" data="data/xml" />
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\23\2d1a5ede-b2c8-432e-bf87-af01a75ebd2d.data" name="stderr" data="data/xml" />
    </result>
    <result type="exec" identifier="exec #2" outcome="CompletedSuccess">
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\23\7f9f9bfe-2d5d-43b1-be1f-a61303701b72.data" name="stdout" data="data/xml" />
      <output file="D:\Open Source\CruiseControl.NET 2.0\project\console\bin\Debug\Test\Artifacts\23\1b65cf49-4902-441f-848b-c2f22234d6b7.data" name="stderr" data="data/xml" />
    </result>
  </result>
</build>

Now it is easy to see which tasks ran, what the outcome of the tasks were and what output they generated.

In Conclusion

This (hopefully) concludes the changes to the build engine on the server for the stream-based output. In the next set of posts I will delve into using the new output.

Stay tuned…

A Quick Review: Task Classifications

A few posts ago (here), I classified the tasks into how they generate output. Broadly, the output can be split between the build log or directly generated to the file system. There is also a third type of output – external output. This is output that is generated by an external application and stored directly to the file system.

These three types all need to be modified to generate index entries, so the results can be accessed later on. Tasks that write to the build log (e.g. via strings in the old version) or those that write to streams (e.g. direct output to the file system) are easy enough – we already have methods on TaskContext to  generate indexed streams. However external applications that generate files directly need consideration.

So, for this post I am going to convert two tasks, one that generates build log data and the other that manipulates external data. The tasks I will convert are:

• ExecutableTask
• NDependTask

This is also the order of complexity of the conversion I omitted a task from the internal generation, as those tasks often have additional (and complex) logic. I will hopefully look at them in a future post.

General Execution: ExecutableTask

One of the most basic tasks in the system is the executable task. This task is used for executing any external application, which also makes it one of the most versatile tasks. This task attempts to execute the application, nothing more, nothing less. Any output to standard out or standard error is captured by CruiseControl.NET and written to the build log.

In the current version of CruiseControl.NET (1.5 or earlier), generating the build output from the executable task is a two stage process. First, everything is written to a StringBuilder and then dumped into a string. In the second stage, this string is converted into XML.

For my first version of the conversion to streams, I did a similar process – except using streams. The process executor would write the standard out and standard error to a stream. Once this had finished, it would then go through and convert the results into XML. This is what I used to check the proof of concept was working.

However, this was still a two stage process – generate the results and format them. There is no reason why this is required – it should be possible to generate the results directly as XML – which is what I have done.

Most of the changes to handle this were done in ProcessExecutor. Previously I had modified the signature for Execute() to take in two streams – one for standard out and the other for standard error. These streams were then wrapped in StreamWriter instances and passed onto the runnable process. When the external application wrote to standard out or standard error, the output was sent to the corresponding StreamWriter, which sent the data to the stream.

My new change was to introduce a new class – XmlStreamWriter. This class wrapped a StreamWriter and overrode some of its methods. The most important method it overrides is WriteLine() – when this method is called it wraps the line in an XML block and ensures the data is valid (e.g. has no invalid XML characters). It also appends a line number and the date/time, just to make the output more meaningful. The other methods that I overrode were to start/end the XML document (since an XML document can have only one root) and to clean up when necessary.

Now, all ExecutableTask does is start two result streams and pass these through the levels to ProcessExecutor. It also tells the logic to wrap these streams in the new XmlStreamWriter class, which does the automatic conversion. And that’s it – nice and easy. the infrastructure does the rest of the work – generating the index entries, writing them to the log file, ensuring the output is XML, etc.

Analysing Code: NDependTask

The NDependTask calls an external application called NDepend. This application performs a static analysis on a codebase and produces a number of statistics about the code, including complexity, cyclic redundancy and interrelationships. The reason I choose this task is it generates a number of files which need to be included in the build results. These files include both XML data and non-XML data (e.g. images). Additionally, since it is an external task it generates standard out and standard error to be included.

The first part of the conversion is easy – this is almost identical to ExecutableTask – initialise the streams for standard out and standard error, pass them through the layers to where the application is actually executed and store the result for later.

It’s the second part that requires some additional changes. After the external application has finished, the task checks for any new files that were generated. These files are then imported into the results.

To handle this I added a new method to TaskContext – ImportResultFile(). This method has a similar method signature to CreateResultStream() but with two important differences – it has a filename argument and it does  not return anything. This method simply generates a new index entry and moves the file to the results folder. Now external files can be included in the results index.

A Simple Path

Now we have a nice simple migration path for tasks to use the new index. For tasks that generate data they can use the CreateResultStream() method on TaskContext. This will generate a new stream for the data, plus the associated index entries. These streams can then be passed into ProcessExecutor for handling the stdout and stderr output.

For tasks that generate files via an external application, the ImportResultFile() method can be used. This moves the external file into the data folder, plus generates the associated index entries. This also offers the choice to move or copy the files so the calling task can tidy up if necessary.

These two methods, either separate or together, handle most of the data generation for the current tasks/publishers. There are a few tasks/publishers that won’t like this approach, mainly because they need to manipulate files (i.e. read/write access), but as these tasks/publisher tend to be outside the norm I’m going to ignore them for the moment.

The next challenge is to look into accessing these new index entries and the associated files. So stay tuned for a review of how we currently use the build logs and how it will change…

Good Bye XmlLogPublisher

As part of the changes for implementing the task context I am removing XmlLogPublisher. This is to provide a standardised way of generating the log – while still exposing the required functionality to the other tasks that need it.

In the current version of CruiseControl.NET users need to add an xmllogger element to their project configuration (preferably in the publishers section, but there is no reason why they can’t put it in the tasks section). When this task is run it will generate the log file, which can then be used by sequent tasks. However, to complicate things, some tasks generate their own version of the log file (normally in memory) – e.g. StatisticsPublisher and EmailPublisher. Plus if the user adds a publishers section and forgets to add the xmllogger element, then no log will be written!

Note: if the user does not define a publishers section, a publishers section will be generated by default that contains the xmllogger element.

So, to standardise and simplify things, I am moving the functionality from XmlLogPublisher into TaskContext. This functionality will be called automatically by Finialise() in TaskContext (this method is called when an integration finishes).

Currently XmlLogPublisher generates the folder for saving the log in, generates the name of the log file and then writes out the log (via XmlIntegrationResultWriter). I have moved these into TaskContext as separate methods (as GenerateLogFolder(), GenerateLogFilename() and WriteCurrentLog()). The Finalise() method calls all of these, but I have exposed them as public methods so they can be called from other locations.

Additionally, I made a small change to XmlIntegrationResultWriter. Because I want to write the current result entries out via the writer, I changed the constructor to take in an enumeration of TaskResultDetails. I also changed the WriteTaskResults() to output these results (previously it was outputting the data from the tasks). This required changing the statistics and e-mail publishers to use TaskContext instead of XmlIntegrationResultWriter. Unfortunately it also means these are both breaking at the moment, as they are expecting a valid log file to be generated, and it has now been changed to only have index entries! In a future post I will look at modifying these so they work again

Finally to round out this change, I added another new method to TaskContext. This method (GenerateResultsSnapshot()) will generate the current snapshot of all the results. This breaks the normal working of TaskContext, in that it allows a child context to access a property on the parent context. This means that property (the list of index entries) now needs to be locked before it can be accessed. I did consider using the reader/writer lock instead of the general monitor lock, but it sounds like the .NET 2.0 version is flawed, so I decided to avoid it.

So, in saying farewell to XmlLogPublisher, I have moved most of its functionality to TaskContext. I’ve also expanded both TaskContext and XmlIntegrationResultWriter to handle writing the index entries to the log file.

Hello XmlLogPublisher

Having removed most of the functionality from XmlLogPublisher, I rebuilt it to use the new functionality from TaskContext. Why? So it is still possible to generate a build log part way through the build process. This is required for those tasks/publishers that want to do something with a physical build log file. For example, the FTP publisher might want to send the build log to a remote machine, the package publisher might want to include it in a package, etc. The new version allows this functionality.

However, I also expanded some other functionality to make this even easier. As part of the changes in removing the functionality, I removed the log directory configuration setting to Project. While the new XmlLogPublisher still has this setting, I wanted to expose the project-level setting to the publisher. Currently tasks/publishers work in isolation – they don’t know anything about the project that owns them (beyond what is passed about in the IIntegrationResult.) So, I wanted to expose project configuration settings, but in an immutable way.

To do this, I have added a new class called ProjectConfiguration. This class exposes a number of the projects from Project in an immutable form. These are loaded from the project when the class is initialised and are read-only. I then added this new class as a property on TaskContext. Now, any task with an associated TaskContext can access these project configuration settings. At the moment there are only a few settings, but over time I will expand ProjectConfiguration to include most of the configuration settings (yes, it is a work in progress!)

So, now the new XmlLogPublisher can use its own log folder, the project’s log folder (if set) or the project’s artefact folder. At the same time, other tasks can now access project settings in a safe, immutable way.

So, that’s the first set of changes – XmlLogPublisher has been changed to fit the new structure, and the index entries are now being written to the disk. In the next set of changes I’ll look at modifying some tasks/publishers to use the new indexing methods.

Stay tuned…

Data about Data (and so it goes…)

In my last post (here) I reviewed all the tasks and publishers currently implemented. As part of the review there were a number of tasks/publishers identified that generated output that is not currently included in the build log. As part of the changes, we want to be able to handle these files with a standard process as well.

In general, the strategy is to let each task write directly to disk, instead of writing to memory (in a string). It is the task’s job to decide what to write and how. In an even earlier post (here) I introduced the concept of a TaskContext. This context is responsible for starting new streams and storing references to them. While this concept is still valid, it needs a couple of extensions.

My Initial Plan

In my initial plan I was going to let the task choose the file name. The task would call an open stream method on the context and tell the context the file name and a result type name. The context would then open a new file stream using the file name in a folder for the build. If the file name was already in use, the context would generate a unique name (normally by inserting a literal and number into the name). To handle multi-threading, each context required its own folder, and when a context was finished, the owner would then move all the files from that folder into its own folder, again ensuring uniqueness of the names.

This added some extra complexity, just to produce easily readable file names. It did not guarantee that a file name would match a the expected task (i.e. there could be a nunit.xml that did not come from the NUnit task). The task type name was only stored in the index file, and the index file name was not unique (it would overwrite previous versions!)

Additionally, the context only handled generating new streams, it didn’t handle importing files generated externally (e.g. output from external applications). These would still exist outside the generation process, and they were not included in the index file.

Finally, to handle the duo-stream output from external tasks, I had put a stream merging process into place. This would literally take two (or more) streams and merge them together, plus it also converted the streams into an XML format (e.g. one element per line, with a line type per element). Nice maybe, but of questionable value since it adds additionally processing overhead!

Changing the Process

In retrospect I made things harder by trying to be nice and also by having the entire indexing outside of the current way of doing things (since this would also have a minimal impact on the current way of doing things.) Now, I think that was a bad decision – these changes are going to be breaking anyway, so lets try and do things properly!

First, unique file names: the name of the file needs to be unique, no matter which task generates it, whether there are duplicates of a task type, or multiple files of the same name being merged. The uniqueness needs to be enforced even when different threads generate files, and whether the files are generated internally or externally. Rather than inventing a new process for unique names I’ll use a tried and true method – GUIDs. When a new result stream is opened the physical name of the stream will be a GUID – no more, no less.

The index entries for each stream will contain the necessary data to identify what each file is for and the type of data contained. These metadata items are:

• Task type: the type of task that generated the result. This will be the name of the task in the configuration (e.g. nunit, msbuild, nant, etc.) and it will be up to the calling task to pass in.
• Result name: this is the internal name of the file. Examples would be stdout, stderr, test results, etc. Again, it is up to the task to pass in this name and it should be unique per task type (although not sure how to enforce this!)
• Task identifier: this is an identifier for the actual instance of the task. Again, this will come from the calling task and could be something like the user-configured task name, an internally generated identifier or the date/time. Basically, it is used to group streams together within an actual instance of a task.
• Task order: this is the order in which the result was generated. The context will generate this number and associate it with a result. When a child context is merged, the owning context is responsible for ensuring the order numbers are unique. As such this identified the order in which tasks are completed.
• Data type: this is the type of data the file contains. Ideally, this will be the mime type of the data, so it can be used in serving the files via the dashboard. Again, this will need to come from the calling task.

Creating Index Entries

Now that I know what I want to store, and who is responsible for populating streams, the question is how are these index entries and streams generated?

Based on the sources of data, there needs to be two ways of generating the indexes. First, there are the internally generated entries. In this scenario the task will request a stream and pass in the required metadata. The context will generate the stream and at the same time generate an index entry. After this, it is the responsibility of the task to manage the stream (including closing it!)

The second scenario is an externally generated file. In this case the task will tell the context where the file is, plus the required metadata. The context will then import (move/copy?) the file into the build folder and add the index entry. The task does not need to do anything else.

And Finally, Some Context

I’ve written a bit about the context and some of how it works, but I still haven’t covered how it is generated or how it writes the indexes.

An initial context will be generated when a project starts building. At the moment this will be associated with the Project, although later on I may move it (especially when we come to splitting the context from the configuration). As each task is run, a child context is started for the task. The task then runs within this context, generating its own indexes. If the task starts child tasks (e.g. parallel task, sequential task, etc.) it must generate a child context for each child task.

After a task is completed, the associated context is finalised (either by the project or the parent task). Now an important point – each task or project only works with one context, and no other task or project will change that context! This ensures that the context will not be messed up by two tasks trying to modify the context at the same time. There is only one exception to this rule – when a context is finalised. At this time the context is locked so the owning task cannot use it anymore and all the index entries are transferred to the parent context.

If the context does not have a parent context (i.e. it is the project-level context), it will generate the final index. This index will then be included in the standard build log. This will require a slight shift in the way the build log is generated. Currently the build log is generated by the xmllogger task, which means it can be generated at different points in a build. This task will be removed from the available tasks, and the build log generated automatically at the end of the process.

At this point you are wondering how can publishers access the build log? For example, the e-mail publisher may want to generate an e-mail containing the results of the build, the FTP publisher might want to send the log to a remote machine, etc. This will be handled in two ways:

1. Each task context will have access to the current list of indexes. This will need to be accessed in a thread safe way, just in case the user has configured a multi-thread project. Either way, the task context will have access to the entire set of index entries from its parent and all ancestors, all the way to the project context. This means the task will only be able to access index entries from finalised task contexts.
2. At will be possible to generate a build log at any point. These build logs will only be in-memory, but will contain all the current build details, plus the current list of index entries (see point 1. above). At the end of a build this will be the same file that is generated for the overall build log. The new build log structure will replace the build results that currently exist with the index entries.

Time To Work

This describes my plan on how I will implement things. Coming up next, some actual implementation details.

What’s Currently Happening

Before I move onto converting the current tasks from using strings to streams, I thought it would be a good time to review the existing tasks and publishers. Looking at the tasks they can be divided into four categories:

1. Those that don’t generate any output
2. Those that generate output in the build log
3. Those that generate external output
4. Those that generate both

The following tasks do not generate any output:

• ArtifactCleanUpTask
• ConditionalPublisher
• CruiseServerControlTask
• EmailPublisher
• ForceBuildPublisher
• ModificationReaderTask
• NullTask
• ParallelTask
• SequentialTask
• SynchronisationTask

Since these tasks do not generate any output, we don’t have to worry about these tasks (yeah!!)

The second category are those tasks that generate output for the build log. The following tasks fall into this category:

• DevenvTask
• DupFinderTask
• ExecutableTask
• FinalBuilderTask
• GendarmeTask
• MsBuildTask
• NAntTask
• NCoverProfileTask
• NUnitTask
• PowerShellTask
• RakeTask

These are the tasks that I have been primarily concerned with, as they currently generating strings. These strings are then concatenated into one massive string before being written into a file. Since the build log is an XML file, all the strings are XML’enised before they are concatenated. This means if the string is valid XML, it is concatenated directly, otherwise it is wrapped into a CDATA section.

These tasks can work in two general ways. First, they can generate output that is stored in memory as strings (e.g. output from stdout or stderr) – these strings are not persisted to disk at all. Second, they can generate output that is persisted disk, which later gets read into memory and then appended into the log file.

The third category is those tasks that generate individual files directly. These tasks are:

• BuildPublisher
• FtpTask
• ModificationHistoryPublisher
• ModificationWriterTask
• PackagePublisher
• RssPublisher
• StatisticsPublisher
• XmlLogPublisher

These tasks all generate output, but they are responsible for storing the files. They also require custom methods to retrieve the data (normally implemented in CruiseServer and/or Project). Plus, these files can be XML or non-XML (e.g. images, HTML, etc). And as a final complication, these files can read/write – i.e. the task may re-open an existing version of the file, add some more output and save it again.

These tasks open a second area for improvement – providing a standardised way for all output to be stored. But I’ll write more about this soon.

The final category generates both data for the build log and individual data files. These tasks are:

• HttpStatusTask
• MergeFilesTask
• NCoverReportTask
• NDependTask

These tasks basically have both data for the build log and individual data files. The log data is all standard XML, while the non-log data can be anything!

The Strategy for Standardisation

Currently I am changing to use a TaskContext for generating output streams and indexing these streams. This approach can also be expanded to handling the external output files.  There are two main differences between the two: external files can include non-XML data and they can also modify existing files. To simplify things, I decided on to exclude the tasks that modify existing files. These tasks are:

• ModificationHistoryPublisher
• ModificationWriterTask
• RssPublisher
• StatisticsPublisher

The reason why is very simple – the tasks that edit existing files have a set location for the files. These files are managed outside of a build, i.e. they are not build-related. In contrast, the output form the other tasks are specific to the build they were generated in. Once the build has finished, they are never modified (at least in theory).

To handle the XML/non-XML data, I’ll add an extra attribute to the index – the data type. This will be set by the task that generates the index. This will be the standard mime types, although it will involve some work in setting these properly.

Executable Tasks

A second way that tasks can be categorised is whether they execute external application or not. Most (but not all) of these tasks inherit from BaseExecutableTask. To actually execute the application they normally call TryToExecute(), which returns a ProcessResult. The current implementation of this class contains a string containing the standard output and a second string containing standard error. It is then up to the calling task what it does with the results. Currently the results are handling in three ways:

1. The standard output and standard error strings are XML’enised (as above) and concatenated to the build log.
2. The standard error only is XML’enised and concatenated to the build log.
3. Both standard output and standard error are converted into XML lines (all XML reserved characters are converted) and then concatenated to the build log.

The following tasks fall into this category that inherit from BaseExecutableTask:

• DupFinderTask
• ExecutableTask
• GendarmeTask
• MsBuildTask
• NAntTask
• NCoverProfileTask
• NCoverReportTask
• NDependTask
• PowerShellTask
• RakeTask

Additionally NUnitTask calls an external application, but it does not inherit from BaseExecutableTask. This task should also be converted to inherit from BaseExecutableTask, so I will include this task in the list of changes.

The subtle difference between how the output is concatenated means my original approach needs to be revised. Now I’m planning on modifying TryToExecute() so it takes into two streams as arguments, and it is up to the calling task on how these streams will be generated and how they will be processed after the task has executed.

This leaves me with a general outline of how to proceed. Next I will take a look at the approach I’ve implemented and how it affects the existing tasks.

Back to Life, Back to Reality

I have returned from my annual visit to see the in-laws in China, and once again I have worked on some nice goodies to add to CruiseControl.NET (CC.NET).

This time I was looking at two specific areas:

• Converting to streams for results
• A Silverlight RIA

Stream-based Results (Project Ares)

The streams functionality is to try and resolve a long-time issue with running out of memory. Currently CC.NET performs all of its result processing in-memory as strings. This means when a task runs, it generates an in-memory version of the results (e.g. from stdout/stderr or from importing file results). These results are then appended to an ever-increasing copy of the log file, which is finally written out to disk. Now this is fine if 1) the results are small or 2) you have lots of memory, but this can cause problems if these conditions are not met! Additionally the same problem applies not only to generating the results, but retrieving the results!

The solution is simple in concept – instead of writing to memory, write directly to disk instead – hence changing to using streams instead of strings. However like any non-trivial modification, this has a number of far-reaching implications, so it was not as easy as just changing from generating streams to writing to streams.

However, the good news is I got it working, although it still needs a bit of polish to get it working nicely. And the other good news is I documented what I did along the way So over the next few weeks I’ll be reviewing and publishing this documentation on my blog. These posts will be published under Project Ares.

Silverlight Client (Project Capricorn)

The other area I played with was writing a Silverlight 3.0 client for CruiseControl.NET. This was more of a fun project to see what was possible (although it didn’t help that I was offline and had to go by trial and error).

For this project I wanted a similar type of interface to the current dashboard. This allows people to view information at four levels: farm (all monitored servers), server, project and build. More challengingly it allows people to develop their own plug-ins (either via code or XSL-T) and use these within the UI.

As well as implementing a basic dashboard-like UI, I wanted to lever some of the rich functionality that is available in Silverlight – things that are possible in HTML/CSS/Javascript, but are challenging to do.

So, I have put together a very rough implementation of a Silverlight client, although it is very much at a prototype stage. This allows for the basics of the UI (layout, navigation, etc.) plus a plug-in infrastructure for adding new plug-ins. Unfortunately both need work to get up to release level.

So, once I have finished writing about the stream changes I’ll write up about the Silverlight client under Project Capricorn. Hopefully there will be some interest in it, so we can look at completing the project and including it in the official codebase (probably for CC.NET 2.0). Otherwise I’ll move it to the FastForward.NET project and work on it as I have time.

But Wait, There’s More!

Another area that I’ve been slowly working towards for a while now is the ability to make CC.NET distributed. CC.NET as it currently stands has some distributed elements, but it doesn’t really work as a distributed application. Some of the changes I’ve been working on (messaging, hot-upgrades, etc.) have been pieces of the distributed puzzle. The streaming work added a couple more pieces of the puzzle, plus showed a few more challenges to be resolved!

So I’ll be adding a few posts on enhancing CC.NET to be distributed – either some of the issues involved or some of the pieces that have been added. Hopefully by the time we (finally) get to the 2.0 release CC.NET will be able to work as a distributed application

That’s All For Now

So that’s what I’m planning on writing up over the next few weeks. At the same time I’ll be adding the source to SourceForge (under the CCNet2 branch) and hopefully spending some time with the other devs on getting CC.NET ready for the “official” 1.5 release.

Stay tuned…

Yes, It’s That Time Again!

Once again we are coming up to our annual trip to China to see the in-laws. This means I will be disconnected for the next three weeks, but I will have plenty of time on my hands. Last year I worked on a replacement for CCTray (Project Leo) which later became FastForward.NET. This year my plans are just as grand

Currently I am working on converting CruiseControl.NET to use streams instead of strings. Given that this is a big task, with lots of complexity, it’s been slow going. I’m aiming to finish this in my time away – plus look into some other possibilities for the concepts (like distributed builds).

Second, I’m looking at building a Silverlight front-end for CruiseControl.NET. I’ve been playing around with Silverlight at work, so this opens a few possibilities. This will be a standard plug-in for the dashboard, so people can choose to use it or not if they desire. The only downside is it will require .NET 3.5 and Silverlight 3 – so I’m not sure how well this will work on *nix environments.

So, you won’t hear much from me for a wee while, but I hope to have some exciting stuff soon!! Plus my Cantonese should be better by the time I return

Constant Improvement

As well as the core engine for CruiseControl.NET, there are a number of tasks that call external applications. These include tasks that perform:

• Code analysis
• Unit tests
• Run build scripts

And so on. It is very easy to integrate an external application with CruiseControl.NET – all it needs is a command-line interface.

As part of my job I am constantly looking for new tools and utilities that can improve the CI process at my work. When I see new tools or utilities I’ll give them a try. And slowly I build up a list of tools that I think would work nicely with CruiseControl.NET.

One of the acronyms that gets mentioned a lot now-a-days is DRY – Don’t Repeat Yourself. Of course, in theory this is a simple principle, but when the code base starts expanding, it becomes harder and harder to see duplication. To get around this, there are some tools that can be run to detect duplication.

For a long time I haven’t bothered with this analysis as most tools are commercial (and therefore cost money that I don’t have), but not that long ago somebody mentioned DuplicateFinder. This is a little command-line tool that will scan a codebase and report any duplicates.

So I decided to take a break from reducing memory and add in this tool as a new task.

Some Initial Decisions

DuplicateFinder consists of three files:

• The library that performs the analysis
• A command-line application
• A MSBuild task

So the first decision is which should I use? For pure speed, the library is the best approach, but this would require adding a new library the project, one that would only be used for a single task! And the MSBuild task is only useful within a MSBuild script.

So in the end I decided to call the application – even though this will be slightly slower and require an external dependency. But it means if people don’t want this analysis, they don’t have to have the binaries.

DuplicateFinder has three output options:

• Console – text
• Console – XML
• XML file

So, the next decision is which output to add, and in the end I choose console – XML for two reasons. First, CC.NET uses XML for its reporting, so this kept it nice and easy. And writing to the console means there is not an extra file floating around that needs to be cleaned up later.

After these decisions, it was easy to add the new task.

The New Task

The new task is called dupfinder. It has the following parameters:

And here is an example of how to configure it:

<dupfinder recurse="true" width="5" shortenNames="true" includeCode="true" timeout="1200">

  <inputDir>C:\...\Trunk\project\remote</inputDir>

  <fileMask>*.cs</fileMask>

  <executable>C:\...\Trunk\Tools\DuplicateFinder\DupFinder.exe</executable>

  <excludeLines>

    <line>using System.*</line>

  </excludeLines>

  <excludeFiles>

    <file>AssemblyInfo.cs</file>

  </excludeFiles>

</dupfinder>

Some Extras

DuplicateFinder provides a nice simple XML output, but there were a couple of changes I wanted to it. First, the filename contained the entire path of the files. In CC.NET, the full path will point to somewhere on the server, with individual developers potentially having a completely different location.

So I have added the shortenNames parameter to the task. This will load the output XML and trim the filenames. This just iterates through all of the filenames and removes the input path from the name.

Secondly, the output didn’t tell me the code that was duplicated – it just has the starting position and the number of lines. To see the duplicated lines I need to open one of the lines, navigate to the lines and then look.

To get around this I added the includeCode parameter. This will add the lines that were duplicated into the results, so in the dashboard I can easily display them.

The Dashboard

Since this is an analysis type task, I have also added a couple of reports to the dashboard. First, in the project summary there is a summary of the analysis:

Clicking on the “Duplicate Finder Report” will bring up the detailed report:

Yes, nothing fancy, but it does allow someone to quickly see which files have duplicates and which lines are duplicated.

And to make it even easier to install, I have put together a dashboard package that contains the necessary files and settings. All an administrator needs to do is install the package and the report will appear in the dashboard

And Finally, the Why?

Now, any application that can be run from the command-line can be run from a NAnt or MSBuild script, or even via a exec task, so why do I add these new tasks to CruiseControl.NET?

One reason – integration! Adding a task directly to CruiseControl.NET means we can do some nice things with it. For one we can include the progress of the task in the project status – it’s nice to see that we’ve finished the build and are onto unit tests or code analysis, etc. Second, it means only one task – instead of a build task and a merge results task.

And finally, as I have done with the duplicate finder task, we can extend the basic functionality to include new functionality that will add value.

So, if you know of other tools that you think would be nice in CruiseControl.NET, let me know and I can investigate adding them in

Anyone who has installed CruiseControl.NET under Windows Vista or Server 2008 knows that CruiseControl.NET does not like IIS 7.0 (or later). The problem is very simple – IIS 7.0 uses an integrated pipeline for processing ASP.NET requests, while IIS 6.0 uses an ISAPI filter. This simple change introduced a new section in the web.config for integrated applications – system.webServer.

Just to complicate things the dashboard overrides the .aspx extension, basically so it can do it’s own version of MVC. When IIS 7.0 starts up it attempts to validate the configuration. It comes across the new HTTP handlers and basically says “I don’t know what to do!”.

The solution is actually very simple, just add the following section to web.config:

<system.webServer>

  <validation validateIntegratedModeConfiguration="false" />

  <handlers>

    <add verb="*" path="*.aspx" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard" name="CCNetHttpHandler"/>

    <add verb="*" path="*.xml" type="ThoughtWorks.CruiseControl.WebDashboard.MVC.ASPNET.HttpHandler,ThoughtWorks.CruiseControl.WebDashboard" name="CCNetXmlHandler"/>

  </handlers>

</system.webServer>

The validation element tells IIS 7.0 not to validate the old IIS 6.0 settings, while the handlers element adds the required handlers for the dashboard. Simple!

Now, in theory the dashboard should work from an install without anyone needing to change any settings

Introduction

This is the third post in converting CruiseControl.NET from using string to streams. The other posts are:

• Reducing Strings 1: Some Context
• Reducing Strings 2: Getting to the Root

In this post I’m going to take another slight detour from the planned changed. The reason – to see whether the changes are actually working!

The reason why is pretty simple – one of the other devs on the project asked for some proof that my changes would reduce memory usage. Now my rough tests and knowledge of the system said they would, but I thought I should do some proper tests to prove it. And here they are!

Methods

The aim of this set of tests to see whether memory usage is reduced with the new stream-based processing. Since I’ve only converted one task at the memory, all my tests will use that task.

First I put together a small application that would write out “The Preventer of Information Services” to standard output 100,000 times. Using the same test, but writing to a file, generates a ~5 Mb file – not too small and not too big (previous tests on my machine showed I could handle files of up to ~50 Mb within CC.NET).

Next, I put together a simple ccnet.config that would run the application. I then ran this config using the –p switch to execute the project and stop (ccnet -p=Test). This allows me to quickly and easily run just that task without any confounding tasks.

Finally, I profiled the test using ANTS Memory Profiler 5.1 (see credits below).

I performed the same test with both the 1.5 codebase and the new 2.0 “experimental” codebase.

Results

Here are the results from running in 1.5:

This is looking at the private bytes – this is the amount of memory that has been allocated to the process.

And this is looking at the # bytes in all heaps (Gen 0, gen 1, gen 2 and large object).

There is a slow but steady increase in memory for around two thirds of the time, and then there is a huge massive jump in memory usage around the 2:20s mark. There’s also a jump in memory usage right at the end, but as the application is terminating it only appears briefly.

Overall execution time was around 3:12s. Total memory allocated was in excess of 128Mb, while the actual memory usage was fast catching up!

And now, the results from the 2.0 codebase:

There is a huge difference – in both graphs!

The private bytes only reached 32Mb – a quarter the amount of memory allocated for the 1.5 codebase. And the change is even more dramatic for the bytes in all heaps – I had to use the tooltip to find out how much memory was used – a mere 1.31 Mb at the most. Additionally, this high usage point was right at the beginning of the task executable, as opposed to a spike at the end. Finally, the memory usage is actually dropping at the end of the execution in the 2.0 codebase.

Finally, and this surprised me, the execution time was better for the 2.0 codebase (approximately 2:13s). However this might be because my machine is running at near physical memory capacity anyway, so the extra memory needs to come from the paging file.

Analysis

In the 1.5 codebase there are a number of memory jumps. I think these jumps are being caused by different steps in the process of persisting the results to the log file. The slow increase is due to the test task writing to standard output. The first big jump is when the task is completed and the data is being converted into an XML format. The final jump is when the data is actually being merged into the log file for writing to disk.

In contrast, the 2.0 codebase has a much lower memory usage level. The memory level is staying low because the output is being written directly to disk and then freed. The memory manager in .NET can then reuse the memory instead of needing to ask for more.

The second difference lies in the way the results are converted – this is done on a line by line basis in the new code. In the 1.5 codebase everything is done in memory, thus requiring sufficient memory for both the original string and the new string, plus any memory for working space. In the 2.0 codebase the conversion is done on a line by line basis. The line is loaded from the original stream, converted and written to the new stream. Again, a much smaller memory footprint being needed.

Finally, the spike at the end is eliminated as the data is already in the files, so there is no need to merge everything together (although in theory this could still be done.)

Summary

So, in conclusion, the stream changes have significantly reduced the memory footprint – at least for my simple test. Comparing the two codebases, the 2.0 codebase only requires a quarter of the total memory allocated, and internally less than 1% of the old heap allocations! For those machines like mine with only limited physical memory, the changes also result in faster performance!

Credits

I was able to perform these tests thanks to Red Gate. They provided a free software license of their .NET Developer’s Bundle to allow me to do some performance tuning, and I am very thankful to them for this (it turns out they use CruiseControl.NET, so hopefully this will help them too). Their website (http://www.red-gate.com/index.htm) contains the details on their products, plus a lot of useful information on how to use their tools.

For these tests I used the ANTS Memory Profiler, I haven’t tried the other tools in the bundle yet.

I really liked the way you could see what was happening in real-time, it was very easy to see the memory jumps and tie it into what was happening in the application. They also have a whole heap of additional information that is available – like snapshots of how the memory is being used and what has changed. But for this post the graphs had all the information – in future posts I’ll delve into some of the other information available.

So if you are looking for a memory profiling tool, I definitely recommend taking a look at the ANTS Memory Profiler, it takes the guesswork out of what is happening in memory

Continuation

In my last post on this subject (read it here) I added the concept of a task context. This is a context that the task runs within and stores all the output from a task. The next step is to start writing to this context.

One of the core concepts in the context is it generates streams that can be used by the task. The task is then responsible for managing the stream, but the context is responsible for managing the referencing to the stream. So, the trick to generate the streams from the context, pass it through to where they are needed in the tasks and then clean up when the task has finished with the streams.

And of course, that’s where it gets tricky!

The Current Situation

90% of the time, a task does not directly execute an external application. Instead it calls through a number of layers. This allows a number of cross-cutting functions to be built in, but at the same time it makes changes from strings to streams harder.

Here is the current way it works:

The calling task calls the TryToRun() method on BaseExecutableTask, which passes it onto ProcessExecutor and finally RunnableProcess. RunnableProcess internally creates two StringBuilder instances – one for standard output (StdOut) and one for standard error (StdErr). RunnableProcess also provides all the functionality necessary for getting the data from StdOut and StdErr and putting it into these two instances.

When the Run() method on RunnableProcess has finished, it generates a ProcessResult and stores the strings from the two StringBuilder instances into there. From here on, the StdOut and StdErr are stored as strings in memory.

So, the issue now becomes one of where should the streams be initialised? How many streams should be generated? And what do we want to store for future usage?

The New Situation

The main change I am making is removing the instantiation of StdOut and StdErr from within RunnableProcess to BaseExecutableTask. These will be instantiated as streams and passed through to RunnableProcess, where it is an easy enough change to write to the streams instead of a StringBuilder.

Additionally, I’m going to get BaseExecutableTask to generate two streams – one for StdOut and one for StdErr. These get passed down the chain. When the Execute() method on ProcessExecutor has completed, these two streams will be merged into one.

So, this shows how I am changing things:

On a side note, a ProcessResult will still be generated as this contains additional information needed for the tasks (e.g. exit codes, time-out details, etc.)

Based on this plan, most of the work is in the BaseExecutableTask, with minor changes to the other two classes. Additionally, I’m going to do some work around the merging of StdErr and StdOut in TaskContext, as this class is responsible for managing the references.

The Actual Changes

Most of the changes are straight-forward, and reasonably boring (except when I made a mistake and have to debug it!)

RunnableProcess and ProcessExecutor were both modified to accept streams and use them (instead of the internal StringBuilders.) Reasonably simple change – just needed to remember to close the StreamWriters I was using.

BaseExecutableTask got a new override for TryToRun() that uses the new streams functionality. This override includes the task name and type (required for creating a result). Additionally I added a couple of new protected virtual methods methods to allow people to override some of the functionality. This is the creating the result stream and merging results functionality (more on this below).

Finally TaskContext got a new method – MergeResultStreams(). This will literally merge two or more streams into a single stream. It also manages the references – the old references are removed and a new reference is added for the merged file. The merging is handled by a delegate, so individual tasks can define how the results are merged. The default merge is a binary merge – copy all the bytes from each of the streams into a single stream.

BaseExecutableTask defines a custom merge delegate. This merge delegate will merge all the results into an XML format – similar to how ExecutableTask does it currently. However this uses the streams to handle the merging and formatting, instead of directly manipulating strings in memory – so we shouldn’t have the out of memory exceptions

And that’s all there is to it – at least for this phase. At this point the new code breaks a number of other tasks – those that rely on the data being in a single massive string. Plus there are a number of additional overrides that I have added temporarily to reduce the amount of breaking changes.

So, What’s Next?

Looking over my previous what’s next list, I realise I’ve skipped over a couple of points – all of this is included in the task changing. I have modified Project to both generate the task context and associate it with tasks, but I think a bit more work is needed there. Additionally I need to look at the container tasks (e.g. parallel task, sequential task, etc.) to see what needs to change to pass on child contexts.

So, in my next post I plan on covering the “plumbing” for the contexts, and then I’ll return to modifying tasks.

Stay tuned…

Introduction

Earlier this year I added parallel and sequential tasks to CruiseControl.NET (read about them here). These tasks provided the ability to split a project into multi “streams” that ran in parallel. Additionally, projects within CruiseControl.NET already run in parallel – so this functionality has been around for a long time.

However, there are the occasional issue around having tasks running in parallel – and most of the time it revolves around access to a limited resource (e.g. a file on disk, etc.) This opens the possibility for a synchronisation task – which is what I have now added

Within a Project

Normally, within a project tasks run in sequence:

With the parallel and sequential tasks introduced in CruiseControl.NET 1.5, we can now have two (or more) sets of tasks running in parallel:

The problem comes when two (or more) tasks want to access the same resource at the same time, for example writing to a file:

Now if there is a slight timing difference, then this will work (either task 3 finishes before task 6 starts or vice versa). But when task 3 starts before task 6 finishes (or vice versa) then there is a potential resource conflict! The new synchronisation task handles this situation by wrapping the two tasks in a synchronisation context:

Tasks 3 and 6 are both wrapped in an instance of the new synchronisation task. When either of the wrapper tasks start, they check to see if the synchronisation context is free – if it is then the task is allowed to execute. Otherwise it waits until the context is free, and then starts the task.

This means only task 3 or task 6 will be running at any time – the two tasks will not be able to execute at the same time!

Between Projects

The same process also applies for common resources that are shared between projects. In this scenario we have two “streams” of tasks running in parallel:

Again, they might need to access the same resource:

Synchronisation in this scenario works in exactly the same way:

Again, the tasks will need to be wrapped in the new synchronisation task. This task generates the synchronisation context and ensures that only one task can enter the context at any one time.

And the good news is this is the same task for both within project and between projects!

Some Inner Workings

This section covers how the synchronisation task works, so if you don’t care about the technical details, just move onto the next section

Internally the task has a static table that contains a set of named synchronisation objects. When the synchronisation task starts, it first ensures that there is a synchronisation object in the table – if not it will add it. It then uses Monitor.TryEnter() to lock the synchronisation object. When the lock is entered, it then runs all the child tasks in sequence.

There are a few other small details:

• When the lock is entered, the next step is to enter a try/finally clause – this ensures that the lock will be exited, no matter what happens to the child tasks
• The TryEnter() uses a time-out – this can be configured, but defaults to 300 seconds (five minutes)
• The default context is called DefaultSynchronisationContext – the context can have different names

The last point is very important. If the name of the context is not changed for any synchronisation task, then there will only be one synchronisation context for the entire instance of CruiseControl.NET. Changing the context name allows multiple synchronisation contexts (e.g. one per project, one per shared resource, etc.)

Configuring the New Task

This is a wrapper task, similar to the parallel and sequential tasks. This means it is configured in the same way:

  1: <tasks>

  2:   <synchonised timeout="60">

  3:     <tasks>

  4:       <exec>

  5:         <!-- Omitted for brevity -->

  6:       </exec>

  7:     </tasks>

  8:   </synchonised>

  9: </tasks>

This example demonstrates running an exec task within the default context. If the context is not entered within 60 seconds, the task will fail.

  1: <tasks>

  2:   <synchonised context="sharedResource">

  3:     <continueOnFailure>true</continueOnFailure>

  4:     <tasks>

  5:       <exec>

  6:         <!-- Omitted for brevity -->

  7:       </exec>

  8:       <exec>

  9:         <!-- Omitted for brevity -->

 10:       </exec>

 11:       <exec>

 12:         <!-- Omitted for brevity -->

 13:       </exec>

 14:     </tasks>

 15:   </synchonised>

 16: </tasks>

This example shows a named context, with multiple wrapped tasks. All of the tasks will run, whether or not they are successful.

Note: In CruiseControl.NET all simple properties can be set as either attributes or elements. continueOnFailure can be added as an attribute as desired and likewise context could be an element.

The following configuration options are available:

Some Final Words

At the moment this task is only available in the CruiseControl.NET 2.0 branch – which means it won’t be available in the main release for a while. If there is interest in making this task available earlier, let me know and I can move it to the trunk.

Additionally I haven’t spent much time testing this task. There is a full set of unit tests (yes I used TDD), but I haven’t tried it on any “real” instances of CruiseControl.NET yet.

Introduction

This will be the first in a series of posts I’m planning on writing about modifying CruiseControl.NET to be stream-based for the results.

In my previous posts I’ve been looking into reducing memory usage for CruiseControl.NET. Most of the changes have been small tweaks (reducing the number of copies of a string, reducing memory peaking, etc.) However these changes have not addressed the underlying issue – CruiseControl.NET is manipulating some very large strings in memory!

This series will look at changing from using strings for encapsulating task results to using streams. Additionally, I’ll look at splitting our huge massive log file into a series of smaller log files.

Adding Some Context

At the moment, each task is an isolated unit. They exist in their own little worlds, with no interactions with the rest of the project. Information is passed into the task using an IIntegrationResult and information is returned via an IIntegrationResult. Additionally, there are a number of global instances that a task can use (like loggers, file systems, etc.)

For these changes I’m planning on a change in perspective. Instead of a task working in isolation, only communicating via an IIntegrationResult, I’m going to add a task context. When a task runs, it will have this context to call upon.

This means each task can effectively act on its own, while at the same time being part of a bigger picture. Before I explain what I mean by this, let’s take a look at a diagram of how I see this working:

This diagram breaks the task information into two – the configuration and the context. The task configuration is loaded when CruiseControl.NET loads (or is reloaded) and is in theory immutable (should never change unless reloaded). This defines the task, how it works and what it works on. The task configurations sit within the project configuration, and is likewise immutable (although not everything in a project is immutable, more on this in a future post).

So far, this describes how CruiseControl.NET currently works, so no changes needed at all. The new stuff is on the right-hand side of the diagram – context.

Context provides mutable data for a task. This is the data the task uses when it runs, and is reset for every run. Because of this, a task can start with a clean slate, and slowly build up its state within the context.

In the diagram above, I’ve shown two types of context – task context and child context. In the code, they are both the same type (more on this later), but contextually they work in different ways. The task context is the root level context – this matches the project configuration. As each task starts, a child context is started for the task to run in. Then, when the task finishes the child context is loaded into its parent context:

The reason for this design change will (hopefully) become more obvious over time. Basically this provides a “safe” environment for a task to run. It can modify any of its context settings, without needing to worry about screwing up any other tasks. This is especially relevant as we start moving into more complex task types – like parallel execution tasks where each task runs on its own thread!

So we now have the picture where the configuration is immutable and remains unchanged between builds. Instead each build has its own set of context to work with, and each task contributes to this context as it runs.

By this time you’re wondering where are the streams? So, that brings us to our next area:

From Context To Stream

In .NET (and many other languages) a stream is just a sequence of bytes. The Stream class provides a wrapper for whether the bytes are actually stored, which means a stream can be stored anywhere, provide all sorts of extra functionality and can really simplify testing, all without needing to change the code that uses the stream. On the down side, it is up to the developer to provide everything beyond the basic byte manipulation.

This is why behind the context changes I described above. As far as each task is aware, it just interacts with the stream of bytes – writing out the results as they are generated. The context provides the infrastructure around this – where the streams are stored, how to start a stream, merging streams and referencing them for future usage.

When a task needs a stream for writing its results, it calls the context that is is associated with. The context then opens the new file stream (including making sure each file name is unique) and returns the stream instance to the task. The task then uses the stream as desired.

At the same time as opening the stream, the context also stored a reference to the file stream. This is the path to the file that was opened. When the task is finished, the context for the task is merged up into the parent context – including the reference to any files that were opened. When the root level context is finished, it writes all of the file references to an XML index file. This provides the link between the individual task results and the overall build results.

Where To From Here?

This post outlines the basic context changes I’m making to CruiseControl.NET. It allows the generation of streams for each task in a way that is thread safe, plus has the ability to treat results as a whole. I’ve very quickly outlined how the writing process will work, but this still leaves many areas uncovered.

Here is a rough outline of where I’m planning to go from here:

• Add initialising the root context as part of the build process
• Work out how to associate the context with each task
• Change the tasks included with CruiseControl.NET to use the context
• Modify the server interfaces to work with individual task results (instead of the big log file)
• Change the dashboard to use the individual task results
• Fix the build report templates (the XSL-T files) to work with individual results
• Fix the tasks within CruiseControl.NET that use the entire build log (e.g. e-mail task, etc.)
• Add a report generator task that generates and stored the reports on the server (this will use the new HTML report plug-in in the dashboard)
• Add backwards compatibility to the server interface

Somewhere in this process I plan on removing ITaskResult and its associated/dependent classes and methods from the code base. This means that this series of changes will break existing plug-ins! The why behind this decision is two-fold:

1. It simplifies my job – I don’t have to deal with backwards compatibility on the server (I still plan on adding it to the server interfaces so older clients can still connect to CruiseControl.NET)
2. It prevents people in future from going down the strings path when writing new tasks

So that’s all for now, I’ll continue slowly working on this as I have time.

Previously

In the first post in this series (here) I looked at some options for reducing memory usage, in my last post (here) I applied the first option and added caching to the dashboard. This time it is the turn of the server to get some caching.

Again, the main point from the previous posts on caching is the primary interest for most people is seeing why a build has failed. The most recent build logs are expected to be the most used, so caching can help reduce the amount of “copies” of a build log in memory. This is especially relevant when there are a large number of clients (e.g. CCTray) accessing the same build information at roughly the same time.

Quick and Easy

When I added caching to the dashboard, I also added an extra class to handle synchronised access to the data. So basically, all the work has been done already

First, the place where the log is being fetched is the GetLog() method in CruiseServer. I’ve added the method to retrieve the cached log in here – which replaces the previous call (this just retrieved the log from the specified project).

Next, I copied over the RetrieveLogData() method from the dashboard project and changed it to do the old retrieve log call.

Finally, I made one other small modification – I added a lock { } statement to the code. This is to ensure that only one SynchronisedData is ever added for each log file. There is a small possibility of a race condition when the cached data is fetched – because it needs to do a few extra steps if the data is not cached. Adding this lock { } statement completely removes this possibility.

Caching – Tick

So, that’s caching added to both the server and the dashboard. Like I said in my last post, I expect this to average out the memory usage, rather than reducing the average memory level. Hopefully these will work as planned.

If not, I have one final trick up my sleeve – compressing the data. Stay tuned…

Previously

In my last post (read it here) I took a holistic approach to reducing memory usage on the (server) machine. In it I suggested three items to reduce memory usage:

1. Caching on the dashboard
2. Caching on the server
3. Compressing the build logs

In this post I’ll look at implementing the dashboard caching.

To help with the recap, here is a diagram from my last post:

Basically this is stating most of the views to a log will be to the most recent (failed) build, less views to the last successful build and even fewer views for historical builds. This means most of the effort can be focused around caching a very small set of builds.

Introducing Caching

For caching, the first question is what should be cached.

When a request is processed for a build report it goes through this general process:

The request comes in, the process retrieves the log from the server, generates the build request (the actual report) and returns it to the client. I’ve put the generation into a black box, because that’s how I’m going to treat it (it is a complex process within the box). With this simplified view, there are two obvious cache options:

When the process needs a log, it can check the log cache first. If there is a log there, then this log will be used – otherwise it will fetch the log from the server, store it in the cache and then continue.

The second cache is to bypass the entire process – to cache the result and use it. When I investigated the dashboard further I discovered that build reports are already cached (this is via an action proxy – CachingActionProxy.) This proxy action caches each build response into the cache using the raw URL of the incoming request. Since this is working at the moment, I’m not going to change it. I’m not sure how this will work with security in place – so this will need to be investigated.

Before I delve into the implementation, there is an issue that needs to be considered. Retrieving a log can potentially be long-running processes (i.e. it takes more than a few milliseconds to perform.) The last thing we want is multiple users attempting to access a log almost simultaneously and trigger the same process multiple times! Because of this, the design needs to take into account potential race conditions.

Fetching and Caching the Log

The build logs are loaded in BuildRequestTransformer. This is the class within the dashboard process that is responsible for fetching the logs and generating the results – although it passes on these tasks to two very different places.

Within this class there is one method that needs to be modified – Transform(). This method currently looks like:

string log = buildRetriever.GetBuild(buildSpecifier, sessionToken).Log;

return transformer.Transform(log, transformerFileNames, xsltArgs);

Since I am changing the way the logs are retrieved, I’m going to change it to:

var log = this.RetrieveLogData(buildSpecifier, sessionToken);

return transformer.Transform(log, transformerFileNames, xsltArgs);

Of course, all the fun happens in RetrieveLogData() (the rest is just tidying things up.)

So, what happens in RetrieveLogData()? First, a unique cache key is generated. This is the combination of the server, project and build names, plus the user’s session token (if there is one). The session token means we’re going to a per-user cache, but unfortunately it is required since each user could potentially see different information.

Once the key is generated, the method checks if there is already a log under that key – if so it will use that log. If not, it generates a new instance of a SynchronisedData, adds this to the cache and then loads the log from the server.

And that’s all there really is to it – nice and simple.

What about SynchronisedData?

You might have noticed I have added a new class – SynchronisedData (or you might have just assumed that it already existed – I’m good with either.) This class is responsible for ensuring that only one instance of the data exists (or at least tries to.)

Basically, this class allows data to be added to the cache before it is retrieved. Internally it uses a ManualResetEvent to let any other threads know when it is safe to use the data. There is of course a catch – if a thread does not initialise the instance (e.g. retrieves it from the cache), it must call the WaitForLoad() method before retrieving the data – otherwise there might not be any data to use.

Otherwise this class is very simple, it has a single property – Data, plus a couple of methods – WaitForLoad() and LoadData(). WaitForLoad() just makes sure it is safe to use the Data property, while LoadData() allows the caller to load some data into the property.

The caller that initialises the instance is responsible for loading the data via the LoadData() method, all other callers need to use WaitForLoad() to ensure it is safe to use the data.

A Final Word on Caching

Someone might look at just this post and ask why am I suggesting caching as a solution for memory usage? Caching generally increases memory usage, since a process is generating and storing items, instead of just generating and disposing them.

And I have to agree. I expect memory usage to go up on average with these changes. The key phrase in that sentence is “on average”. The basic problem I think is happening is we are getting memory spikes. A number of people are trying to access the same log at the same time – therefore the server and the dashboard are generating multiple copies of the same log and then generating multiple copies of the same report.

Caching solves this problem by exposing the same log and the same report to all users, rather than a copy of the log and report. It does open some possibilities on race conditions (hence the locking code) and also increases average memory usage. But it should (hopefully) stop out of memory exceptions – which can really ruin someone’s day

And a Final Comment

There was a comment on my last post about clearing the cache when the configuration changes. For the log files, this is not an issue as it will be the same log (always, no matter what, logs NEVER change!) But, the build reports might change. Since this is handled by an already existing handler, I’m not sure whether it will handle config changes (my guess is it won’t.)

In my next post I’ll look at making some changes to the server, so stay tuned…

The Problem

This is a continuation on my earlier post on memory issues (read it here). In that post I made some minor changes to reduce memory usage (mainly around string handling). However those changes didn’t fix the underlying problem – CruiseControl.NET is working with large strings in memory.

Now normally the OS and .NET will allow us to play around with large objects in memory without too many issues. Normally…

Sometimes it is possible to use up all the available memory (including swap files, etc.) Why? Because we put too many big strings into memory – and that’s exactly what CruiseControl.NET does. And to make things worse, CruiseControl.NET likes to compound things by the way it works.

So, how does CruiseControl.NET work?

First off, we have the server. This is basically a polling application – every five seconds it checks if a build can start and if so, triggers the build (actually it’s a little bit worse than that, each project has it’s own thread which polls every five seconds, but that’s another matter altogether.) As the build runs it builds up a set of task results, which then get merged together into one big log file:

So, it is possible to have multiple projects, each having multiple large task results in memory. Then, when these get merged into the build log, that’s even more memory used. But wait, there’s more!

As well as the project threads, the server is also responsible for serving results to anyone who requests one of these build logs. This is done by loading the log into memory and then returning the entire log (via .NET Remoting) to the client:

So, as well as having multiple task results and multiple build logs in memory, it is also possible to have multiple instances of the build logs – one per request handler (although in theory these would be quick requests and shouldn’t hold onto memory.) And yes, there’s still more!

The CruiseControl.NET server only allows .NET Remoting clients – which isn’t very helpful if you need to go through a firewall. Plus, people need some way of seeing the results from the tasks – CCTray doesn’t display them. So to handle these situations there is the Web Dashboard. This functions as a client application and does all sorts of things – including manipulating build logs in memory (normally an XSL-T transform):

And where does the Web Dashboard normally sit? On the same machine as the server (unless you’ve changed the default install and installed the dashboard on a separate machine).

So, this gives us a picture of having large strings in memory in several places. The project threads might be okay by themselves, but when you add the communications side, it is very easy to start consuming large amounts of memory with big build logs. Especially when there are a sizeable number of people trying to get results from a build (and what’s the first thing people do when they see a broken build – they go to the server to see what’s broken!)

Pruning Some Memory

Now that we’ve quickly reviewed the possible places where build logs are stored, what can we do to reduce the amount of memory?

Two initial approaches come to mind:

1. Reduce the size of the log file
2. Reduce the number of times each log file is loaded into memory

Option #1 can be done either in CruiseControl.NET or externally. First, we can split the mega-log into smaller logs, second we could write less data, third we could compress the data.

I’ve been looking at what I can do to split the mega-log file into build result specific log files, so a client could just grab the data they need (e.g. just the NAnt/NUnit/MSBuild/etc. results) instead of everything at once. Unfortunately this is a MASSIVE change, that will need to modify both the server and the clients in a large number of places. So for the moment I have spun off a “play” branch in the Subversion repository where I will attempt to do this (oh, and resolve all the associated problems with having multiple result files.) So this is a no-go for the current release.

Next, writing less data – well CruiseControl.NET doesn’t put that much CruiseControl.NET-related data into the log files. Most of the data comes from the external tools. So if this needs to be done, then people need to reduce the amount of data the tools generate – hence this becomes an external task.

So, that leaves compressing the data. Personally I like this approach as it would also reduce the amount of network traffic. Unfortunately there are a couple of gottas – there is no guarantee that compression would actually save space and it requires changes to the clients to decompress the data. But I still think the idea has merit.

Returning back to the two initial approaches, the other approach is to reduce the number of times each log is loaded. This is the approach I want to investigate some more.

Log Viewing Usage Patterns

From my experience with CruiseControl.NET, there are two general usage patterns:

1. THE BUILD HAS BROKEN – what’s wrong?
2. What happened to an older build?

And typically the second pattern also happens a lot as part of pattern #1 when people try to find out what has changed to cause the build to break. Very rarely do people go and look at a historical build.

The following picture sums up this pattern, with the size of the arrow indicating how much usage a build log would get:

So, most people only view the logs when something is broken and they primarily focus on the log of the build that actually failed. A smaller set of people would also check the previous successful build to see if there is anything pertinent to the failure. Finally a very small set of people might go through older logs to see what has happened (your manager is checking up on the amount of work you’ve been doing, etc.)

This sounds like a very good scenario for caching. Since there is a (potentially) small amount of data, this could be cached on the server, the dashboard or even both.

Bring on the Cache(s)

On the server side, we could just cache the log files – although we wouldn’t want to cache too many big ones.

The dashboard offers us a few different possibilities for caching. First, we could take the same approach as for the server and cache the the log files. Second, we could cache the parsed XML documents (since the entire document must be loaded and parsed before it can be transformed). Finally, we could cache the transformed output.

However, like anything in the dashboard, the way the build logs are transformed is not a simple process. Instead we have a number of interfaces and their implementations to go through. The build report generation is a plug-in, which generates multiple actions. Each action can have one or more style sheets, which actually get loaded and processed by a class in the Core library.

To Be Continued…

I’ve gone back and reviewed the memory usage scenarios for CruiseControl.NET with a view of reducing the usage on the server. This took a step back and viewed the wider picture, with a view of all the items that CruiseControl.NET is doing on a (server) machine. Unlike my previous post, this takes into account multiple users doing similar things at the same time.

I’ve come up with a couple of ideas for reducing memory usage – caching and compression – and both of them have some gottas that need to be investigated.

So rather than making this an even longer post, I’m going to stop at this point and look into caching and compression in some future posts.

So stay tuned, more fun to come

Recently I’ve been exploring a few options for refactoring CruiseControl.NET to handle the out of memory exceptions that we have been getting. Unfortunately a lot of the paths I’ve investigated have turned out to be red-herrings – they either have wide-reaching repercussions or they involve significant changes to the server. I’ve even trying profiling the application using dotTrace, without finding any obvious areas for improvement.

So, it’s time to get a little more drastic. Since we are aiming to release the 1.5 version sometime this year (hopefully), I have started a new “experimental” branch. In this branch I’m planning on looking at some more drastic refactoring to resolve the memory issues, plus a couple of other ideas I want to try out for a “CruiseControl.NET 2.0”.

And in case you are wondering, here are some of the ideas I want to try out:

• Distributed builds: being able to take a single project and distribute it over multiple machines
• Build agents/build distributions: extend CruiseControl.NET so there can be master/slave instances, with the master being responsible for distributing build requests across multiple machines
• Data storage layer: move all of the file I/O into a common layer – this is in preparation for adding database persistence

Now, these are just ideas at the moment – with my current amount of free time it’ll be a while before any of these see fruition. But I’ll be doing my standard document things as I go, so if you are wondering why I am writing about these things, this is why.

It’s been a while, I’ve been side tracked with trying to track some performance issues with CC.NET. I have just posted the binaries for the fourth beta – although most of the changes have been around for a few weeks. The following items have been added/fixed:

• Swapped Ok and Cancel buttons
• Cleaned up the settings dialog so the name of the tab is not on each button within the tab
• Added a double-click action to the all projects grid – the user can choose which action to perform
• Added servers and projects to the system tray – clicking on a project triggers the double click action

I am running this as my CC.NET monitor and it seems to be working. I know there are a couple of issues with configuration that need to be resolved, otherwise it is ready for release

The binaries can be downloaded from https://www.ohloh.net/p/FastForwardNET/download.

I’ve been having a number of issues with SourceForge – mainly with its performance. And I’m not the only one – one of the other developers for CruiseControl.NET has been so frustrated that he has set up his own Subversion server, together with a Trac instance.  Additionally, he has been kind enough to let me host FastForward.NET.

So, everything has been moved from SourceForge. I will keep an eye on SourceForge for any issues that people may raise, but I won’t be updating it at all.

So, here are the links for FastForward.NET:

• Trac (issue tracking and documentation): http://dev.nauck-it.de/projects/show/fastforwardnet
• Subversion (source code): http://svn.nauck-it.de/fastforwardnet
• CruiseControl.NET (CI Server): http://build.nauck-it.de/server/build.nauck-it.de/project/FastForward.NET/ViewProjectReport.aspx

Many thanks to Daniel Nauck for letting me use his servers.

The Problem Defined

I’ve been spending a bit of time trying to resolve a number of outstanding issues in JIRA about running out of memory with CruiseControl.NET:

• CCNET-819
• CCNET-955
• CCNET-1659

These are all different examples of getting an OutOfMemoryException when performing a build. Basically a task generates a large output, which CC.NET then attempts to merge into the standard build result. Unfortunately some of these issues have been around a long time (CCNET-819 was first raised in Jan 2007!) which implies that this is a fairly deep rooted issue.

This post contains what I’ve found out so far, and some of my changes to try and reduce the memory usage. Unfortunately, I say “try” as this is both a hard problem to replicate and a hard problem to resolve!

Some Investigation

Looking at the stack traces, the basic issue is with strings. CC.NET is loading the entire build log into memory and manipulating it. Even worse, it can be getting various parts of the results and manipulating them, before writing them into the build log.

When a task executes, it can generate multiple ITaskResult instances. These instances have a Data property which is a string. As you can imagine, this lead to the data being loaded into the various implementations and sticking around until the build has finished. So, if a task generates a 20Mb output, this is added to memory and held for the entire remaining duration of the build. Actually, it’s probably held even longer, as the memory will not be released until a garbage collection is performed.

But, don’t worry, things are even worse! As ITaskResult is an interface, there are a number of different implementations. The implementations I have found are:

• DataTaskResult
• FileTaskResult
• ProcessTaskResult

DataTaskResult is the simplest of the three – it just provides a backing field for the property that contains the string. When this class is initialised the string is loaded and held there until the result is cleaned up. While this is the default result generated, from what I can see in the code it is not actually used (except in the null task.)

FileTaskResult is a view onto a file. Typically a task will generate a file (e.g. when an external application is called) and then this result type is generated to reference the file. Now for the bad news – when this result is generated it opens the file and loads the entire file into memory! So if the task generates a huge file (e.g. NCover results for a large code base, etc.) the file is loaded into memory and hangs around like a bad smell

The final result, ProcessTaskResult, is the most complex of the three. It is also the cause of one of the issues. When an external task is executed it normally uses the ProcessExecutor class. The output of this class is a ProcessResult, which contains all the output written to StandardOutput and StandardError. And yes, these are both stored as strings. ProcessTaskResult manipulates these strings to generate the final output, and that’s where the problems start coming in.

Some Background

If you are wondering why I am picking on strings at this point here is some background. In .NET strings are immutable. Once a string has been allocated it cannot be changed. But what about the string manipulation functionality? Unlike C++, these functions do not modify the string, instead they generate a new instance of the string (with the modifications of course), which is then another immutable string in memory.

So for example, if we had the string “This is a test   ” and we wanted to remove the extra whitespace we would call string.Trim(). At this point we now have two strings in memory: “This is a test   ” and “This is a test”. Even if we assigned the new string to the old string variable, this is still the case.

So, when is the old string removed from memory? When garbage collection occurs. So on a heavily loaded machine where garbage collection is running slowly, these strings can hang around for a while.

Of course, for my short example, this isn’t really a problem – most machines could hold millions of strings like these without any problems. But, imagine if the string is 20Mb in length (this isn’t too out of the ordinary for some processes). All of a sudden the memory will be chewed up very quickly (especially as the OS likes to take a fair chunk).

At this point, I should mention most OSs now-a-days can use disk swapping to extend the amount of free memory. However problems occur if the OS is unable to find a large enough continuous free space to allocate – this is typically what is causing the out-of-memory errors. RAM has been filled and the OS is unable to swap out some memory.

So, short of planning around with garbage collection (which I have no intention of even attempting), our best approach is to reduce the amount of strings we are generating.

Starting Small

As I mentioned earlier, FileTaskResult loads the entire file into memory when the class is initialised. The first change is load the file only when it is needed, and not store a reference to the string. Garbage collection works by detecting whether an object has been orphaned – that is whether there are any references to the object. If there are not references, then garbage collection will remove it (at least that is my understanding).

So now file results will only be loaded when they are needed, and then disposed of as soon as possible (again depending on garbage collection). This gives us a little more to manoeuvre. But, it’s only the tip of the ice berg.

Strings upon Strings upon Strings

Looking at the way ProcessTaskResult works, and how the instances are generated, there is a lot of strings being generated.

As an example, in ExecutableTask it needs to check if there is any output from the executable. This output can be from either standard out or standard error. The literal line is:

if (!StringUtil.IsWhitespace(processResult.StandardOutput + processResult.StandardError))

This combines the two outputs together to generate a new string – hence twice the memory allocation. So, this can be changed to:

if (!StringUtil.IsWhitespace(processResult.StandardOutput) || !StringUtil.IsWhitespace(processResult.StandardError))

which means the original strings are used instead of generating a new string.

Next, digging a little deeper, this is how StringUtil.IsWhitespace() works:

return value.Trim() == 0;

As you can see, this is generating another string! If the string is whitespace, then there is no overhead – it will just generate an empty string, no matter how large the original was. But, if the original had 20Mb of non-whitespace, there is now a second 20Mb string generated!

I’ve replaced this approach with a slightly more complex approach. First I check if the string is null or empty – if it is then the string is considered whitespace. If it is not null or empty, then the new routine iterates through every character in the string and checks if the character is a whitespace character (using char.IsWhiteSpace()). if the character is not whitespace, the loop is exited and false is returned. Otherwise, it will continue through the entire string and return true if no whitespace characters are encountered.

I’m not sure on the performance loss with this change, but I imagine the string data type will be doing something similar with its Trim() method, so it shouldn’t be too bad. Plus this has the advantage of not generating a new string – hence lower memory usage.

There is a third change that I am thinking about, but I haven’t done yet. When the ProcessTaskResult is generated, the caller often calls StringUtil.MakeBuildResult(). This converts the original newline delimited string into an XML structure with an element per line. However I’m not sure exactly whether this will provide any improvements, so I’ve left it for the moment.

Baby Steps

These are my first few baby steps to reducing memory usage in CC.NET. At the moment I’ve just been looking at the server. My initial steps have been to reduce the memory usage by reducing the number of strings generated and held in memory. While I’m hoping this will reduce memory usage, I don’t think it will have too much of an impact (big sigh).

The real problem, and the massive challenge, is to remove the strings from memory as much as possible.  I have tried looking into using streams, but this will be a massive change

Second, this is only looking at the server side. Two of the issues are with the dashboard – which is a whole different area to look into.

Anyway, baby steps, I’ll continue looking into what can be done to improve the memory usage.

Happy anniversary! Yes, my first post on this blog was one year ago! That was when I first adventured into the wonderful world of open source and embarked on enhancing and improving CruiseControl.NET.

Since then a lot of things have changed! Here are some of the highlights:

• I have a son (he’s almost five months old now!)
• I’ve written 178 posts
• I’ve changed 60 thousand lines of code (wow!)
• I’ve added a new application to CC.NET (CCValidator)
• And I’ve started my own open source project (FastForward.NET)

In terms of CruiseControl.NET, I’ve added the following functionality:

• Security
• Dynamic build values and parameters
• Message-based communications
• Common communications client
• Monitor API
• NDepend and NCover tasks
• Image and HTML support in the dashboard
• Packages
• Hot-swapping
• Parallel and sequential tasks
• Task statuses

Plus I’ve gotten to know a great bunch of people, and learnt a lot about coding and development.

So, all in all, one year down the line, I think it has been very much worth it

Here’s to another year!!!