The Problem Defined
I’ve been spending a bit of time trying to resolve a number of outstanding issues in JIRA about running out of memory with CruiseControl.NET:
These are all different examples of getting an OutOfMemoryException when performing a build. Basically a task generates a large output, which CC.NET then attempts to merge into the standard build result. Unfortunately some of these issues have been around a long time (CCNET-819 was first raised in Jan 2007!) which implies that this is a fairly deep rooted issue.
This post contains what I’ve found out so far, and some of my changes to try and reduce the memory usage. Unfortunately, I say “try” as this is both a hard problem to replicate and a hard problem to resolve!
Some Investigation
Looking at the stack traces, the basic issue is with strings. CC.NET is loading the entire build log into memory and manipulating it. Even worse, it can be getting various parts of the results and manipulating them, before writing them into the build log.
When a task executes, it can generate multiple ITaskResult instances. These instances have a Data property which is a string. As you can imagine, this lead to the data being loaded into the various implementations and sticking around until the build has finished. So, if a task generates a 20Mb output, this is added to memory and held for the entire remaining duration of the build. Actually, it’s probably held even longer, as the memory will not be released until a garbage collection is performed.
But, don’t worry, things are even worse! As ITaskResult is an interface, there are a number of different implementations. The implementations I have found are:
- DataTaskResult
- FileTaskResult
- ProcessTaskResult
DataTaskResult is the simplest of the three – it just provides a backing field for the property that contains the string. When this class is initialised the string is loaded and held there until the result is cleaned up. While this is the default result generated, from what I can see in the code it is not actually used (except in the null task.)
FileTaskResult is a view onto a file. Typically a task will generate a file (e.g. when an external application is called) and then this result type is generated to reference the file. Now for the bad news – when this result is generated it opens the file and loads the entire file into memory! So if the task generates a huge file (e.g. NCover results for a large code base, etc.) the file is loaded into memory and hangs around like a bad smell 
The final result, ProcessTaskResult, is the most complex of the three. It is also the cause of one of the issues. When an external task is executed it normally uses the ProcessExecutor class. The output of this class is a ProcessResult, which contains all the output written to StandardOutput and StandardError. And yes, these are both stored as strings. ProcessTaskResult manipulates these strings to generate the final output, and that’s where the problems start coming in.
Some Background
If you are wondering why I am picking on strings at this point here is some background. In .NET strings are immutable. Once a string has been allocated it cannot be changed. But what about the string manipulation functionality? Unlike C++, these functions do not modify the string, instead they generate a new instance of the string (with the modifications of course), which is then another immutable string in memory.
So for example, if we had the string “This is a test ” and we wanted to remove the extra whitespace we would call string.Trim(). At this point we now have two strings in memory: “This is a test ” and “This is a test”. Even if we assigned the new string to the old string variable, this is still the case.
So, when is the old string removed from memory? When garbage collection occurs. So on a heavily loaded machine where garbage collection is running slowly, these strings can hang around for a while.
Of course, for my short example, this isn’t really a problem – most machines could hold millions of strings like these without any problems. But, imagine if the string is 20Mb in length (this isn’t too out of the ordinary for some processes). All of a sudden the memory will be chewed up very quickly (especially as the OS likes to take a fair chunk).
At this point, I should mention most OSs now-a-days can use disk swapping to extend the amount of free memory. However problems occur if the OS is unable to find a large enough continuous free space to allocate – this is typically what is causing the out-of-memory errors. RAM has been filled and the OS is unable to swap out some memory.
So, short of planning around with garbage collection (which I have no intention of even attempting), our best approach is to reduce the amount of strings we are generating.
Starting Small
As I mentioned earlier, FileTaskResult loads the entire file into memory when the class is initialised. The first change is load the file only when it is needed, and not store a reference to the string. Garbage collection works by detecting whether an object has been orphaned – that is whether there are any references to the object. If there are not references, then garbage collection will remove it (at least that is my understanding).
So now file results will only be loaded when they are needed, and then disposed of as soon as possible (again depending on garbage collection). This gives us a little more to manoeuvre. But, it’s only the tip of the ice berg.
Strings upon Strings upon Strings
Looking at the way ProcessTaskResult works, and how the instances are generated, there is a lot of strings being generated.
As an example, in ExecutableTask it needs to check if there is any output from the executable. This output can be from either standard out or standard error. The literal line is:
if (!StringUtil.IsWhitespace(processResult.StandardOutput + processResult.StandardError))
This combines the two outputs together to generate a new string – hence twice the memory allocation. So, this can be changed to:
if (!StringUtil.IsWhitespace(processResult.StandardOutput) || !StringUtil.IsWhitespace(processResult.StandardError))
which means the original strings are used instead of generating a new string.
Next, digging a little deeper, this is how StringUtil.IsWhitespace() works:
return value.Trim() == 0;
As you can see, this is generating another string! If the string is whitespace, then there is no overhead – it will just generate an empty string, no matter how large the original was. But, if the original had 20Mb of non-whitespace, there is now a second 20Mb string generated!
I’ve replaced this approach with a slightly more complex approach. First I check if the string is null or empty – if it is then the string is considered whitespace. If it is not null or empty, then the new routine iterates through every character in the string and checks if the character is a whitespace character (using char.IsWhiteSpace()). if the character is not whitespace, the loop is exited and false is returned. Otherwise, it will continue through the entire string and return true if no whitespace characters are encountered.
I’m not sure on the performance loss with this change, but I imagine the string data type will be doing something similar with its Trim() method, so it shouldn’t be too bad. Plus this has the advantage of not generating a new string – hence lower memory usage.
There is a third change that I am thinking about, but I haven’t done yet. When the ProcessTaskResult is generated, the caller often calls StringUtil.MakeBuildResult(). This converts the original newline delimited string into an XML structure with an element per line. However I’m not sure exactly whether this will provide any improvements, so I’ve left it for the moment.
Baby Steps
These are my first few baby steps to reducing memory usage in CC.NET. At the moment I’ve just been looking at the server. My initial steps have been to reduce the memory usage by reducing the number of strings generated and held in memory. While I’m hoping this will reduce memory usage, I don’t think it will have too much of an impact (big sigh).
The real problem, and the massive challenge, is to remove the strings from memory as much as possible. I have tried looking into using streams, but this will be a massive change
Second, this is only looking at the server side. Two of the issues are with the dashboard – which is a whole different area to look into.
Anyway, baby steps, I’ll continue looking into what can be done to improve the memory usage.
RSS - Posts
