Fun with Concurrency

As part of my studies I am researching multi-processor (or concurrent) programming. One potential issue in concurrent programming is two threads (or processors) attempting to access the same variable at the same time. When both threads are reading, this is not an issue, but if one (or both) are writing this can lead to corrupted data being retrieved.

For example if one thread wrote the value 7 to a variable. Later, at the same time another thread write the value –3 to the same variable at the same time as the original thread is writing. The question is what would the thread get? We would hope it would get either 7 or –3, but it is equally likely it could get something like –7, a corruption combining the two values!

After reading this, I wondered if this was possible in CruiseControl.NET. And the answer is yes – it is very easy to get something like this!

Each project runs on its own thread, but as long as it only updates its own data it is ok. But, any requests for information come in via .NET Remoting (e.g. if the dashboard wants to update its view or CCTray is polling the project status). When the request is received CruiseControl.NET handles it on a separate thread from the project thread. If the project is also updating at the same time the read is happening, there would be data corruption!

To test this, I modified the code slightly. I added some extra tracing statements – basically to show the start and end of GetProjectStatus() and every call to LastIntegration. I also modified the integration status so it has a unique ID that changes every time the status is updated (this is a GUID). I then ran some tests to see if this ID changed while GetProjectStatus() was running.

What I was looking for is very simple, if the ID changed, then GetProjectStatus() will be returning corrupted data. After a number of attempts (plus adding a small Sleep into LastIntegration) I was able to get the following:

[20:DEBUG] Start => GetProjectStatus()
[20:DEBUG] Get ===> LastIntegration: a9ba4e605ac1476099e447eaf1175bb1
[20:DEBUG] Get ===> LastIntegration: 840b18e5a5244ebfbbe4066557b0c3a6
[20:DEBUG] Get ===> LastIntegration: 840b18e5a5244ebfbbe4066557b0c3a6
[20:DEBUG] Get ===> LastIntegration: 840b18e5a5244ebfbbe4066557b0c3a6
[20:DEBUG] End ===> GetProjectStatus()

The first Get has a different ID from all the rest – which means it is a different LastIntegration. This shows that the status can have corrupted data (i.e. data from multiple sources!) Now I had to cheat a little to get this, but that fact that it is possible means it can happen! Hopefully though it is very unlikely to happen!

Going forward, what does this mean?

At the moment, I’m not sure. It does raise the possibility of some really subtle bugs in the communications, but whether this is significant, I don’t know. Since no one has mentioned it, it probably doesn’t happen much, but we should probably look into it sometime.

And of course the other question is: what other areas are potentially affected like this?

The joys of concurrent programming…