Improved NCover Integration

Is Your Code Covered?

One of the important metrics for TDD is code coverage – how much of the code base is actually being tested by unit tests. It’s not the only metric (even with 100% coverage you can’t guarantee that everything will be perfect), but it is a reasonably easy metric to measure.

There are various tools out there to measure this, but one of my favourites is NCover. NCover originally started as an open source tool, and has now moved into the commercial realm. I originally didn’t see the point in measuring code coverage, but since I’ve been using NCover at work, I now realise it is invaluable. And I certainly think they’ve improved things a lot with NCover 3.

If you want to find out more about NCover, they have a website with all the details – http://www.ncover.com/. They have even put together a pretty good tutorial on integrating CruiseControl.NET with NCover – http://docs.ncover.com/how-to/continuous-integration/cruisecontrol-net/.

However, the integration between the two products isn’t really that good. I should stress, it’s not because of NCover – they do everything right – it’s because CruiseControl.NET has a way of working, and we’ve so far forced everyone into using that path.

But the good news is this is changing!

The Current State

Currently CruiseControl.NET doesn’t directly handle running NCover – it has to be run from the <exec> task, or as part of a build script (e.g. <nant>, <msbuild>, etc.) Next, the results from NCover need a second task to merge them – otherwise they are ignored!

Finally, CruiseControl.NET uses XSL-T files to convert the results (which must be in XML) into HTML. So, if there is a huge amount of data, the result displays are very, very slow.

So, there are a number of hoops to jump through for getting NCover results into CruiseControl.NET. A much better approach would be some simple tasks to directly integrate and display the results – including NCover’s HTML reports.

And What’s New

As an alternate approach to using <exec> or build scripts, I have added two new tasks to CruiseControl.NET. These tasks allow CruiseControl.NET to run NCover directly and automatically merge the results!

Why two tasks? This is because of the way NCover works – there is a profiling application and a reporting application. Originally I disagreed with this approach, but since I’ve played around with it, I’ve come to agree with this approach. With the two applications, you can profile a project once and then generate multiple different reports. And if you have a huge suite of tests, it’s certainly good news to run the profiler only once!

Before I move into a tutorial on these tasks, I should mention that these tasks are designed to run with NCover 3.0. The profiling task might work with NCover 2.0, but I certainly haven’t tested it (nor will I), and the reporting task certainly won’t!

A Quick Tutorial

So, how do you run NCover from CruiseControl.NET?

First, you need to make sure NCover is installed and your code is built. Additionally you’ll need to know the location of both NCover and the unit test runner executables, and also the location of the binaries. Once you know this this, then you’re all ready to start!

Running NCover

Now, open ccnet.config (the default location of this is in the server folder in the install location). If you’ve already used CruiseControl.NET, you should have one or more projects set up – otherwise take a look at the documentation on how to do this (http://confluence.public.thoughtworks.org/display/CCNET/Configuring+the+Server). So you should have something that looks like this:

<project name="MyProject">
    <webURL>http://mybuildserver/ccnet/</webURL>
    <workingDirectory>C:\Integration\MyProject\WorkingDirectory</workingDirectory>
    <artifactDirectory>C:\Integration\MyProject\Artifacts</artifactDirectory>
    <!-- <triggers /> -->
    <!-- <sourcecontrol /> -->
    <tasks>
        <!-- Build task here, e.g. <nant>, <msbuild>, etc. -->
        <!-- other tasks here ... -->
    </tasks>
    <publishers>
        <xmllogger />
    </publishers>
</project>
To add NCover profiling, add an <ncoverProfile> task to the tasks block. I would recommend adding this after the test binaries have been built so you are profiling the latest version. As a bare minimum you need to specify the locations of NCover, your test runner and the binaries.
Your project will then look something like the following:
<project name="MyProject">
  <webURL>http://mybuildserver/ccnet/</webURL>
  <workingDirectory>C:\Integration\MyProject\WorkingDirectory</workingDirectory>
  <artifactDirectory>C:\Integration\MyProject\Artifacts</artifactDirectory>
  <!-- <triggers /> -->
  <!-- <sourcecontrol /> -->
  <tasks>
    <!-- Build task here, e.g. <nant>, <msbuild>, etc. -->
    <!-- other tasks here ... -->
    <ncoverProfile>
      <executable>C:\Program Files\NCover\NCover.Console.exe</executable>
      <program>tools\nunit\nunit-console.exe</program>
      <testProject>myproject.test.dll</testProject>
      <workingDir>build\unittests</workingDir>
      <includedAssemblies>myproject.*.dll</includedAssemblies>
    </ncoverProfile>
  </tasks>
  <publishers>
    <xmllogger />
  </publishers>
</project>
This tell CruiseControl.NET to run NCover from C:\Program Files\NCover\NCover.Console.exe. The test runner is nunit-console (this is located in C:\Integration\MyProject\WorkingDirectory\tools\nunit\nunit-console.exe). The project that will be tested is myproject.tests.dll (this could be any valid nunit project).
This will run NCover and automatically merge the results. You could stop at this point and just use the old NCover XSL-T files to display the results.
However, NCover also comes with an impressive suite of reports, which be also be run directly. To do this you need to add one or more <ncoverReport> tasks. Each report task can run one or more reports – this allows you to change the parameters.
For example, to produce a summary report with a minimum coverage of 95%:
<ncoverReport>
  <executable>C:\Program Files\NCover\NCover.Reporting.exe</executable>
  <outputDir>ncover\reports</outputDir>
  <reports>
    <report>Summary</report>
  </reports>
  <minimumThresholds>
    <coverageThreshold metric="SymbolCoverage" value="95"/>
  </minimumThresholds>
</ncoverReport>
Or to generate a full report that is ordered by coverage percentage in a descending order:
<ncoverReport>
  <executable>C:\Program Files\NCover\NCover.Reporting.exe</executable>
  <outputDir>ncover\reports</outputDir>
  <reports>
    <report>FullCoverageReport</report>
  </reports>
  <sortBy>CoveragePercentageDescending</sortBy>
</ncoverReport>
The full details on what can be reported on is available from the NCover site (http://docs.ncover.com/ref/3-0/ncover-reporting/).
Coming back to our example project, I’ve added the default report (full report):
<project name="MyProject">
  <webURL>http://mybuildserver/ccnet/</webURL>
  <workingDirectory>C:\Integration\MyProject\WorkingDirectory</workingDirectory>
  <artifactDirectory>C:\Integration\MyProject\Artifacts</artifactDirectory>
  <!-- <triggers /> -->
  <!-- <sourcecontrol /> -->
  <tasks>
    <!-- Build task here, e.g. <nant>, <msbuild>, etc. -->
    <!-- other tasks here ... -->
    <ncoverProfile>
      <executable>C:\Program Files\NCover\NCover.Console.exe</executable>
      <program>tools\nunit\nunit-console.exe</program>
      <testProject>myproject.tests.dll</testProject>
      <workingDir>build\unittests</workingDir>
      <includedAssemblies>myproject.*.dll</includedAssemblies>
    </ncoverProfile>
    <ncoverReport>
      <executable>C:\Program Files\NCover\NCover.Reporting.exe</executable>
      <outputDir>ncover\reports</outputDir>
    </ncoverReport>
  </tasks>
  <publishers>
    <xmllogger />
  </publishers>
</project>
Again, this will automatically merge the results for you. Additionally the reports are also stored in the ncover/reports folder (this will be in the C:\Integration\MyProject\Artifacts folder).
And that’s all that is required to run NCover in CruiseControl.NET! Nice and easy, but still allowing the full power of NCover.
Displaying Reports
As well as using the XSL-T reports (from the NCover tutorial) you can also directly display the HTML reports within the dashboard.
To display a report within the dashboard is very easy. First open dashboard.config (the default location of this is in the webdashboard folder in the install location). Next find the <buildPlugins> and add an <htmlReportPlugin> element for each report to display.
For example, to display the full coverage report your <buildPlugins> section would look like the following:
<buildPlugins>
  <buildReportBuildPlugin>
    <xslFileNames>
      <xslFile>xsl\header.xsl</xslFile>
      <xslFile>xsl\modifications.xsl</xslFile>
    </xslFileNames>
  </buildReportBuildPlugin>
  <buildLogBuildPlugin />
  <htmlReportPlugin description="NCover Report" actionName="NCoverBuildReport" htmlFileName="ncover\fullcoveragereport.html" />
</buildPlugins>
And that’s all that is required – no need to copy any files! This is what it will look like:

As you can see, this is the full report from NCover. In this case, the report is actually a frameset with sub-reports! Clicking around within the report works just like expected. Additionally, if you don’t have enough screen space, you can open up the report in its own window (so you don’t have all the CruiseControl.NET stuff around it.)
You can add as many <htmlReportPlugin> elements as desired – just make sure the description and the actionName are different for each.
One Final Note
This new functionality is all part of the 1.5.0 release currently. This means if you are using 1.4.4 or earlier, it won’t work. Since this is still in development, I have included the full reference for the new tasks below. The binaries and installer can be downloaded from http://ccnetlive.thoughtworks.com/CCNet-builds/1.5.0/.
So feel free to try this functionality – if you have ay problems with it, either leave me a comment or post a message on the mailing list.
Task Reference
This is the full reference for the new tasks. Since these tasks map through to the NCover executables I have also included the mappings for the command-line arguments. The full details of these arguments are available at http://docs.ncover.com/ref/3-0/ncover-reporting/command-line.
ncoverProfile






Name
Description
Type
Required
Default


program
The program to execute and collect coverage statistics from
String
Yes
n/a


testProject
The project that contains the tests. If relative, this will be relative to baseDir
String
No
None


programParameters
The parameters to pass to the program.
String
No
None


executable
The executable to use.
String
No
Ncover.Console


description
A description of the task.
String
No
None


timeout
The time-out period in seconds. If the task does no finish running in this time it will be terminated.
Int
No
600


baseDir
The base directory to use. All relative parameters will be relative to this parameter.
String
No
Working directory for the project


workingDir
The working directory for the executable. If relative, this will be relative to baseDir.

Maps to //w
String
No
baseDir


publish
Whether to publish the results or not.
Boolean
No
True


logFile
The location of the NCover log file. If relative, this will be relative to baseDir.

Maps to //l
String
No
None


logLevel
The profiler log level.

Maps to /ll
Enum (Default, None, Normal, Verbose)
No
Default


projectName
The name of the project.

Maps to //p
String
No
None


coverageFile
The location to write the coverage file to. If relative, this will be relative to baseDir.

Maps to //x
String
No
Coverage.xml


coverageMetric
The coverage metric to use.

Maps to //ct
String
No
None


excludedAttributes
The attributes to exclude.

Maps to //ea
String
No
None


excludedAssemblies
The assemblies to exclude.

Maps to //eas
String
No
None


excludedFiles
The files to exclude.

Maps to //ef
String
No
None


excludedMethods
The methods to exclude.

Maps to //em
String
No
None


excludedTypes
The types to exclude.

Maps to //et
String
No
None


includedAttributes
The attributes to include.

Maps to //ia
String
No
None


includedAssemblies
The assemblies to include.

Maps to //ias
String
No
None


includedFiles
The files to include.

Maps to //if
String
No
None


includedTypes
The types to include.

Maps to //it
String
No
None


disableAutoexclusion
Whether to turn off auto-exclusion or not.

Maps to //na
Boolean
No
False


processModule
The module to process.

Maps to //pm
String
No
None


symbolSearch
The symbol search policy to use.

Maps to //ssp
String
No
None


trendFile
The location to write the trend file to. If relative, this will be relative to baseDir.

Maps to //at
String
No
None


buildId
A custom build id to attach.

Maps to //bi
String
No
The build label


settingsFile
A location to read the settings from. If relative, this will be relative to baseDir.

Maps to //cr
String
No
None


register
Temporarily register NCover.

Maps to //reg
Boolean
No
False


applicationLoadWait
The amount of time NCover will wait for the application to start up.

Maps to //wal
Int
No
None


iis
Whether to cover IIS or not.

Maps to //iis
Boolean
No
False


serviceTimeout
The time-out period for covering a service or IIS.

Maps to //st
Int
No
None


windowsService
A Windows service to cover.

Maps to //svc
String
No
None



ncoverReport






Name
Description
Type
Required
Default


executable
The executable to use.
String
No
NCover.Reporting


description
A description of the task.
String
No
None


timeout
The time-out period in seconds. If the task does no finish running in this time it will be terminated.
Int
No
600 


baseDir
The base directory to use. All relative parameters will be relative to this parameter.
String
No
Working directory for the project


workingDir
The working directory for the executable. If relative, this will be relative to baseDir.

Maps to //w
String
No
baseDir


coverageFile
The location to read the coverage date from.
String
No
Coverage.xml


clearFilters
Should the coverage filters be cleared.

Maps to //ccf
Boolean
No
None


filters
The coverage filters to apply.

Maps to //cf
Array of coverageFilter
No
None


minimumThresholds
The minimum coverage thresholds.

Maps to //mc
Array of coverageThreshold
No
None


minimumCoverage
Whether to use minimum coverage or not.

Maps to //mcsc
Boolean
No
False


xmlReportFilter
The type of report filtering to use.

Maps to //rdf
Enum (Default, Assembly, Namespace)
No
Default


satisfactory
The satisfactory coverage thresholds.

Maps to //sct
Array of coverageThreshold
No
None


numberToReport
The maximum number of items to report.

Maps to //sct
Int
No
-1


trendOutput
The file to output the trend information to. If relative, this will be relative to baseDir.

Maps to //at
String
No
None


trendInput
The file to import the trend information from. If relative, this will be relative to baseDir.

Maps to //lt
String
No
None


buildId
A custom build id to attach.

Maps to //bi
String
No
The build label


hide
The elements to hide.

Maps to //hi
String
No
None


outputDir
The directory to output the reports to. If relative, this will be relative to baseDir.
String
No
None


reports
The reports to generate.

Maps to //or
Array of Enum (FullCoverageReport, Summary, UncoveredCodeSections, SymbolSourceCode, SymbolSourceCodeClass, SymbolSourceCodeClassMethod, MethodSourceCode, MethodSourceCodeClass, MethodSourceCodeClassMethod, SymbolModule, SymbolModuleNamespace, SymbolModuleNamespaceClass, SymbolModuleNamespaceClassMethod, SymbolCCModuleClassFailedCoverageTop, SymbolCCModuleClassCoverageTop, MethodModuleNamespaceClass, MethodModuleNamespaceClassMethod, MethodCCModuleClassFailedCoverageTop, MethodCCModuleClassCoverageTop)
No
FullCoverageReport


projectName
The project name to use in the reports.

Maps to //p
String
No
None


sortBy
The sort order to use.

Maps to //so
Enum (None, Name, ClassLine, CoveragePercentageAscending, CoveragePercentageDescending, UnvisitedSequencePointsAscending, UnvisitedSequencePointsDescending, VisitCountAscending, VisitCountDescending, FunctionCoverageAscending, FunctionCoverageDescending)
No
None


uncoveredAmount
The amount of uncovered items to report on.

Maps to //tu
Int
No
None


mergeMode
The merge mode to use.

Maps to //mfm
Enum (Default, KeepSourceFilters, Destructive, AppendFilters)
No
Default


mergeFile
The file to store the merged data in. If relative, this will be relative to baseDir.

Maps to //s
String
No
None



coverageFilter






Name
Description
Type
Required
Default


data
The pattern to use for matching items,
String
Yes
n/a


type
The type of item.
Enum (Default, Assembly, Namespace, Class, Method, Document)
No
Default


regex
Whether this is a regex or not.
Boolean
No
False


include
Whether to include items or not.
Boolean
No
False



coverageThreshold






Name
Description
Type
Required
Default


metric
The coverage metric.
Enum (SymbolCoverage, BranchCoverage, MethodCoverage, CyclomaticComplexity)
Yes
n/a


value
The minimum coverage value
Int
No
None


type
The type of item.
Enum (Default, View, Module, Namespace, Class)
No
None


pattern
The matching pattern to use.
String
No
None