Archive

Posts Tagged ‘Web Dashboard’

Categorized Dashboard

One of the issues that is on our issues list is how to visualise  large numbers of projects. While the main suggestion is currently out of our reach (having sub-projects) there was another suggestion that has a lot of merit – modifying the dashboard to allow grouping projects by their category. And even better, there was an associated patch Smile 

So now there is the option to show a grouped dashboard:

image

This displays the projects in their categories – making it easy to see which projects belong together. The band at the top shows the name of the category (its currently blank if there is no name) and the colour gives an indication of the status of the projects within the category. The categories can be collapsed to save space (actually when the view is first shown all the categories are collapsed):

image

This is a separate plug-in, so it can be used together with the primary dashboard, or it can replace the primary dashboard (this is what is happening in my examples.)

Replacing the primary dashboard is easy, open up dashboard.config, find the farmReportFarmPlugin element and set the categories attribute to true:

 <farmReportFarmPlugin categories="true" /> 

To have both styles of dashboard, leave the categories attribute as false and add a new element to the farmPlugins element:

 <categorizedFarmReportFarmPlugin /> 

Hopefully this plug-in will make it easier to see what is happening in the dashboard. Thanks to Ian Glover for providing this patch Open-mouthed smile

Categories: CruiseControl.Net Tags:

A New Version of NDepend

30 November, 2010 1 comment

Revisiting NDepend

Last year I put together a few code changes to improve the integration of CruiseControl.NET and NDepend. NDepend is a commercial product for performing static analysis of code. It produces a raft of pretty graphs showing problem areas and can run rules to detect common problems.

At the time I put together a simple task that called the command-line version of NDepend to simplify adding NDepend analysis to a build. I also made a couple of modifications to the dashboard (and the server) to allow displaying images generated from within the build. Together these made it reasonably easy to use NDepend within a CI process.

Fast forward a few months and there is a new version of NDepend!

One of the new features of NDepend 3 is the ability to generate a rich HTML report of the analysis. This includes more information than the original analysis, plus since the HTML is pre-generated it reduces the workload on the server as there is no need to run the XML through an XSL transform. So far, so good, but the question is how does it integrate with the dashboard? Again there is good news – the dashboard was modified in CC.NET 1.5 to allow the display of HTML reports in an iframe (for NCover.) So in theory there is no need for changes to use the new reports in CC.NET Smile

Unfortunately, as so often is the case, theory did not much reality Sad smile

While there was no need to change the dashboard side, the task was not correctly working. The mechanism we were using within CC.NET to detect new or changed files was not picking up all the files! This means when the report was being displayed certain files were missing, causing the report to look horrible. So after a few e-mails with Patrick from NDepend we settled on a suitable solution for the problem. This solution means in future when NDepend changes there will be no need to change CC.NET again Smile Now that’s my type of solution!

However this does mean that if you are using NDepend then we recommend you use a recent version of CC.NET (1.6.7737 or later).

Configuring CC.NET with NDepend

Configuring CC.NET to use NDepend is reasonably easy. There are two required steps.

First to run an NDepend analysis in CC.NET requires adding an <ndepend> element to a project. This can be as simple as:

 
<ndepend>    
  <executable>path to NDepend.Console.exe</executable>    
  <project>path to ndproj file</project>  
</ndepend> 

This will run the analysis and store the required files in the artefacts folder. While this task has been slightly modified it still has the same configuration options from CC.NET 1.4.4 onwards.

Second, to display the report in the dashboard an <htmlReportPlugin> element needs to be added to the build plug-ins. If the default settings for NDepend are being used the following element needs to be added:

 
<htmlReportPlugin description="NDepend" actionName="ndepend" htmlFileName="NDepend\NDependReport.html"/> 

The key part here is the htmlFileName attribute – this needs to point to the report file within the artefacts.

Once these two changes are in place then you can run an NDepend analysis and see the result in the dashboard:

image

This report uses JavaScript to allow some rich interactions, and they all work perfectly fine within CC.NET Smile

Hopefully this will make it even easier to analysis your code using NDepend and CC.NET.

Categories: CruiseControl.Net Tags:

SubMain’s CodeIt.Right and CruiseControl.NET

A New Tool

I’m always on the look out for developer tools that can help my development. And if the tools can be integrated with CruiseControl.NET – even better!

Recently I found another static code analysis tool – but one with a difference. The tool is called CodeIt.Right by SubMain (http://submain.com/products/codeit.right.aspx). This performs a static analysis for C# and VB.NET code. The difference is it also offers automatic refactoring – it can fix the problems that it finds!

I got in touch with SubMain and they were happy for me to integrate it with CruiseControl.NET. So, I have put together an integration package to include it.

The Task

As well as integration with the IDE, CodeIt.Right comes with a command line version. This tool goes through and analyses the code and produces an XML report out the other side. So it was a simple matter to add a new task that calls the command-line version.

Rather than stopping there, I’ve tried to add a bit of extra value. First, the task will automatically merge the XML results file. Since this is integrated into the task itself, it will only merge the file if the task is run. I mention this because it is often an issue with the merge task – if it is put in the publishers section it will always merge the file, whether or not the task that is supposed to generate it ran or not.

Second, After the analysis has finished, the task can be configured to scan the results file and fail if any violations over a severity level have occurred.

To call the task is as simple as:

<codeItRight>
  <project>remote.csproj</project>
  <executable>Tools\SubMain\CodeIt.Right\SubMain.CodeItRight.Cmd</executable>
</codeItRight>

Where <project> is the project to analyse (you can also use <solution> for scanning an entire solution) and <executable> is the location of the command line executable. A more complete example is:

<codeItRight>
  <solution>ccnet.sln</solution>
  <executable>Tools\CodeIt.Right\SubMain.CodeItRight.Cmd</executable>
  <crData>ccnet.crdata</crData>
  <failureThreshold>Error</failureThreshold>
  <profile>FullScan</profile>
  <reportingThreshold>Warning</reportingThreshold>
  <timeout>600</timeout>
</codeItRight>

Dashboard Integration

The second part of this integration is to add a dashboard plug-in. Since the report is in XML, it is a simple matter to convert it to a report.

CodeIt.Right already comes with an XSL-T template to generate a report, but since it is part of the product, I wasn’t able to include it in CruiseControl.NET. Instead I have produced a different report. Initially I tried to duplicate the CodeIt.Right report, but my XSL-T skills weren’t up to the challenge :-( So, instead I have produced a CruiseControl.NET specific report.

This report is similar to the main project grid – it displays all the results in a sortable grid:

image

Additionally, the results can be filtered, thus reducing the “noise” in the report.

I also added a summary report, which is included on the main build report:

image

Finally, to round everything off I built a dashboard deployment package, so adding the new reports is as simple as going to the admin console and installing it.

In Conclusion

This was a fairly easy tool to integrate into CruiseControl.NET. The task simply runs the tool and merges the result, with the added benefit of checking the results to fail the build if necessary. The reports were just some simple XSL-T templates, plus a package to make it easy to install!

Sorting the Grid with “Locked” Rows

27 January, 2010 3 comments

A Simple Goal

In the dashboard there are two types of rows: the project details and the build status. For 1.5 we had removed the build status row and instead attached it as a tool-tip. However, there were a number of complaints about this – people didn’t like the way we (read I) implemented the tool tips.

So, after a quick trip back to the design board, I decided to put the build status back in as a static row. However this is where the problem starts – the project grid in 1.5 was changed to use client-side sorting (save on the round-trips to the server.) Just adding the build status gives some weird results:

image

Notice that the build status says “Setup for documentation generation” and the project that is actually building is the CCNet-Doc project. The build status line should be associated with the CCNet-Doc project, not the CCNet project! If multiple projects are building, it will put all the build statuses at the bottom of the grid (or the top if the grid is sorted in reverse order!)

So, how can this be fixed?

Some Background

Before I move onto how I resolved the problem, first some background.

Since this is client-side functionality, the actual sorting is done via JavaScript. Rather than re-inventing the wheel, we use jQuery and the tableSorter plug-in by Christian Bach (http://tablesorter.com/docs/index.html). This provides the basic functionality for sorting with a small footprint.

However this plug-in does not have the ability to “lock” rows together. In Christian’s defence, I’m not sure how it could be done in a generic way.

But, not all is lost. As well as providing the basic functionality the tableSorter plug-in also exposes an extension system to allow people to manipulate the grid after it has been sorted. This is what I used to “lock” the rows together.

The HTML

In order to make sense of the code below, here is an example of the HTML used to generate the above table:

<table class="SortableGrid" id="StatusGrid">
    <thead>
        <tr class="ProjectGridHeader">
            <!-- Header definition here -->
        </tr>
    </thead>
    <tbody>
        <tr id="projectData1">
            <!-- Project details definition here: CCNet -->
        </tr>
        <tr id="projectData2">
            <!-- Project details definition here: CCNet-Doc -->
        </tr>
        <tr class="buildStatus" id="link2">
            <!-- Build status definition here: CCNet-Doc -->
        </tr>
    </tbody>
    <tfoot>
        <tr>
            <!-- Footer definition here -->
        </tr>
    </tfoot>
</table>

I have omitted the details of each row – for the code below it is not required.

The important parts to note are:

  • Each <tr> in the <tbody> element has an id tag.
  • For project details rows the id starts with “projectData”
  • For build status rows the id starts with “link”
  • For linked rows the number in the id is the same (e.g. 2 in the example above)
  • All the build status rows have a class of “buildStatus”

This is all generated by the nVelocity template on the server.

Some jQuery Goodness

Now to lock the rows together, all we need is a bit of jQuery and some knowledge on the tableSorter plug-in.

To add an extension to the plug-in is a simple matter of adding a widget. This is done at a global level and then associated with the table when the tableSorter is initialised.

Here is the widget that I added:

$.tablesorter.addWidget({
    id: 'statusDisplay',
    format:function(table){
		$("tr.buildStatus",table.tBodies[0]).each(function(){
		    var row = $(this);
		    
		    // Get the identifier of the owning row
		    var id = row.attr('id');
		    var linkedId = 'projectData' + id.substring(4);
		    var parent = $('#' + linkedId);
		    
		    // Move the row to after the owning row
		    row.insertAfter(parent);
		});
    }
});

This selects gets all the rows with the “buildStatus” class and iterates through each row doing the following:

  1. gets the id of the row
  2. calculates the id of the linked row
  3. gets the linked row
  4. moves the status row underneath the details row (insertAfter will move the row)

All fairly simple with jQuery.

The final part is to associate the widget with the actual tableSorter instance. This is done when the tableSorter is initialised. I used the extensibility model of jQuery so I could separate the code from the template:

$.fn.initialiseProjectGrid = function(config){
    // Initialise the configuration
    var defaultConfig = {
        sortList: [[0,0]]
    };
    config = $.extend(defaultConfig, config || {});
    
    // Initialise the sorting
    this.tablesorter({
        sortList: config.sortList,
        widgets: ['statusDisplay'] 
    });
    
    // Allow chaining
    return this;
};

This ensures that the configuration is set, then initialises the tableSorter and associates the widget. To actually use the function is as simple as:

$('#StatusGrid').initialiseProjectGrid();

And that’s all there is to it.

In Summary

Here’s what the grid now looks like:

image

The status row is now underneath the project details where it should be. Sorting on the different columns will correctly keep the status row under the project it is for.

This also show how easy it is to extend the tableSorter plug-in to add some extra functionality. Thanks to Christian for making it so easy :-)

Beware Excessive Caching

Recently I added a few tweaks to CC.NET to try and reduce the memory usage (read the posts here and here). Unfortunately, this had the opposite affect to what I had hoped and instead increased memory usage!

The reason why is simple – I was too aggressive in my caching. I had added caches to both the dashboard and the server, and set the sliding expiry time to one hour. While this worked fine with smaller log files, it absolutely killed the dashboard for larger log files.

I tested on my machine with a 45Mb log file (big yes, but not outside the realm of possible). With the above settings I could view six log files – no more! Additionally, this locked the memory for an hour, both in the server and the dashboard (double memory usage – *ouch*)

Coming back to the original goal, it was to smooth out the peaks and dips in memory usage so it is a more even approach. So I’ve cut back on the amount of caching.

First, I reduce the amount of time log files were cached on the server. In my original scenario I was predicting that most people were interested in the most recent build, beyond that they were less interested. So I cut the caching time right back to five minutes (this is the default polling time within CC.NET for the interval trigger.)

And since I realise that I don’t know what other people have, I have made this a configurable setting. If your server has more memory, you can increase the cache time and hold more logs in memory. You can also reduce the time, but this can potentially lead to cache churn, so be careful with no going too low!

Second, I removed caching from the dashboard altogether. This is because the dashboard already caches the build pages, so this was leading to double caching again. Since people are most interested in the build pages rather than the log, I decided it was acceptable to remove the dashboard caching.

The result, I was now unable get out of memory exceptions with the 45Mb build logs on my machine (at least when viewing them in the dashboard). There will still be an absolute limit somewhere, but until we move to streams this is an ok solution.

Of course, to really fix the problem we will change to using streams, but there is no timeframe for these changes yet…

Parlez-vous Français

16 December, 2009 7 comments

A Long Time Ago…

Back in the 1.4 days, when we were still investigating what to put into 1.5, I looked into adding multi-lingual support to CruiseControl.NET. Basically I had hoped to make CruiseControl.NET available to more people, especially those who did not speak English (or for whom English was not their first language).

As nice as the idea was, there was one major snag – no translators :-( So after a while the idea died a death. Unfortunately, my linguist ability is extremely limited (I’m learning Chinese, but it’s a painful and slow process!) So when we started preparing for the 1.5 release, the multi-lingual changes were dropped.

Recently, one of our users asked how to switch languages in the dashboard. When we told him it couldn’t be done, he came back with an offer of translating into French for us! So I have dusted off some of the old archived code and re-added multi-languages to CruiseControl.NET.

Keeping It Simple

Now the original version of the multi-lingual changes started with the Remote project and worked their way up through Core, Server and then onto the Dashboard (CCTray was bypassed). This meant, in theory, an administrator could completely switch their CC.NET install into a different language.

While this was a grand vision, it had a few detractors. For one, should exceptions be translated? (I said yes, others said no.) What about documentation? (Especially considering our documentation needs some TLC anyway.) What about support? Testing? Etc.

So for this second attempt I have pared it right back. The only thing that is translated is the dashboard – everything to do with the actual CI server is untouched. This, hopefully, bypasses most of the above concerns, while still allowing non-English speakers to use the system.

Additionally, I changed the code slightly. Originally the language was hard-coded – everyone had to use the selected language. The new version uses the language detection in ASP.NET and selects the user’s primary language (as specified by their browser).

Now for the bad news – currently there is only one translation, French, and it is still a work in progress. So, if you speak French, here is what the dashboard looks like currently:

image

And just to show that it is different, if I change my language back to English:

image

Acknowledgements

As I mentioned earlier my linguist ability is virtually nil. Louis-Philippe Carignan is the kind volunteer who provided the French translation. Many thanks Louis-Philippe, I hope this helps you :-)

Categories: CruiseControl.Net Tags:

Timeline with Historical Data

10 December, 2009 1 comment

After my last post I was asked if the new timeline plug-in works with existing data. The answer is yes – there were no server changes for this plug-in, it uses the existing server interfaces, it just provides a new UI on top the existing infrastructure.

Just to prove this works, I updated the installation at my work. This installation has been running since Apr 2008 (yes, before I started contributing to CC.NET). here is what the timeline looks like during one of our “busy” periods:

image

Again, I will repeat – this required no server changes! I did completely re-install the web dashboard for this plug-in, since it has a number of additional JavaScript and Image files that the package does not include (this “assumes” they are on the server). When we get around to doing some more work on the dashboard packages we can look at building a full package that includes all the additional files – but for now, I’m lazy ;)

Categories: CruiseControl.Net Tags:

Build Timeline

9 December, 2009 2 comments

What’s Happening?

At my work we are playing around with a JavaScript widget called Simile Timeline (details on the widget are available at http://simile-widgets.org/wiki/Timeline). This is what provides the timeline in sites like http://www.ohloh.net. Basically, it provides a scrollable timeline, with events on it (these can be point events or durations). Clicking on an event will pop up a little “bubble” with extra details in.

Anyway, I thought that this is a nice fit for CruiseControl.NET – after all, builds are events in time. And since I need a change from working on documentation, I thought I’d take a look at what is involved in adding this to the dashboard.

Nice and Easy

Since this is a JavaScript widget, I downloaded the files and added them to the project. Next I added te plugin and action classes to the dashboard.

The action class performs two actions:

The first action is to generate the actual view. This is what the users see, so I added a status bar to provide extra details. This is a plain NVelocity template that generates some HTML. Additionally, there is a <script> section in it that initialises the timeline widget, loads the data and provides the status messages.

The second action is the data retrieval. The widget expects either an XML or JSON data file, so I choose XML (since it is already supported by the dashboard.) This file contains a list of events, the date they happened on, plus a title. For the dashboard, the title is just the status (success or failure), plus the build label.

And that’s all there really is to it. I added a few extra bits and pieces to try and make the user interface a bit more friendly. And like everything else, there is still some room for improvement. Currently clicking on a build just shows the date/time, the title plus a link to go to the view. In future it would be nice to have some extra details in this pop-up, but this will require some work on the back-end.

The Results

Here’s what the timeline view looks like:

image

Clicking on an event pops up the bubble:

image

And of course, the timeline can be scrolled:

image

That’s All Folks

So, that’s all there is to this plug-in. Nice and simple.

I personally like being able to see a timeline of events, what do other people think? If there is interest in this I’ll add it to the 1.5 release, otherwise it’ll wait until some undefined point in the future :-)

Categories: CruiseControl.Net Tags:

Debugging WebDashboard

One of the current issues with debugging WebDashboard is a developer needs to attach to the IIS process under which it is running, which assumes the developer has the right version of Visual Studio. Yuck!

Visual Studio 2005 and later comes with its own web server, which can be used for debugging. This is as easy as setting the web project as the start-up project and pressing F5 (or using the appropriate menu command). Everything else is handled automatically for the developer.

Unfortunately this doesn’t work for WebDashboard – instead an error pops up saying a DLL cannot be started directly. The reason for this history – CruiseControl.NET was originally a .NET 1.0 project and it has slowly been migrated over time.

So, why can’t we use the built-in web server? Because the project file has not been configured as a “web application” (does anyone remember those annoying web site projects?), so Visual Studio doesn’t know how to use the web server.

The good news, is this is very easy to change! While I was away I had a situation where I wanted to debug the dashboard, but couldn’t (I was having issues with IIS 5.0 on my laptop). So I poked around in the project configuration a bit and found this line:

<ProjectType>Local</ProjectType>

Looking at .NET 2.0 web projects (or later) they have a different line:

<ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{fae04ec0-301f-11d3-bf4b-00c04f79efbc}</ProjectTypeGuids>

Things have changed a bit between 1.1 and 2.0 – and obviously the upgrade wizard couldn’t handle it.

So, replacing the first line with the second is all that is needed to upgrade to a proper web application, including being able to use F5 debugging.

Nice and simple, I really should have looked into this earlier!!

I’m Back: Holiday’s Over, Time to Write!

17 November, 2009 Leave a comment

Back to Life, Back to Reality

I have returned from my annual visit to see the in-laws in China, and once again I have worked on some nice goodies to add to CruiseControl.NET (CC.NET).

This time I was looking at two specific areas:

  • Converting to streams for results
  • A Silverlight RIA

Stream-based Results (Project Ares)

The streams functionality is to try and resolve a long-time issue with running out of memory. Currently CC.NET performs all of its result processing in-memory as strings. This means when a task runs, it generates an in-memory version of the results (e.g. from stdout/stderr or from importing file results). These results are then appended to an ever-increasing copy of the log file, which is finally written out to disk. Now this is fine if 1) the results are small or 2) you have lots of memory, but this can cause problems if these conditions are not met! Additionally the same problem applies not only to generating the results, but retrieving the results!

The solution is simple in concept – instead of writing to memory, write directly to disk instead – hence changing to using streams instead of strings. However like any non-trivial modification, this has a number of far-reaching implications, so it was not as easy as just changing from generating streams to writing to streams.

However, the good news is I got it working, although it still needs a bit of polish to get it working nicely. And the other good news is I documented what I did along the way :-) So over the next few weeks I’ll be reviewing and publishing this documentation on my blog. These posts will be published under Project Ares.

Silverlight Client (Project Capricorn)

The other area I played with was writing a Silverlight 3.0 client for CruiseControl.NET. This was more of a fun project to see what was possible (although it didn’t help that I was offline and had to go by trial and error).

For this project I wanted a similar type of interface to the current dashboard. This allows people to view information at four levels: farm (all monitored servers), server, project and build. More challengingly it allows people to develop their own plug-ins (either via code or XSL-T) and use these within the UI.

As well as implementing a basic dashboard-like UI, I wanted to lever some of the rich functionality that is available in Silverlight – things that are possible in HTML/CSS/Javascript, but are challenging to do.

So, I have put together a very rough implementation of a Silverlight client, although it is very much at a prototype stage. This allows for the basics of the UI (layout, navigation, etc.) plus a plug-in infrastructure for adding new plug-ins. Unfortunately both need work to get up to release level.

So, once I have finished writing about the stream changes I’ll write up about the Silverlight client under Project Capricorn. Hopefully there will be some interest in it, so we can look at completing the project and including it in the official codebase (probably for CC.NET 2.0). Otherwise I’ll move it to the FastForward.NET project and work on it as I have time.

But Wait, There’s More!

Another area that I’ve been slowly working towards for a while now is the ability to make CC.NET distributed. CC.NET as it currently stands has some distributed elements, but it doesn’t really work as a distributed application. Some of the changes I’ve been working on (messaging, hot-upgrades, etc.) have been pieces of the distributed puzzle. The streaming work added a couple more pieces of the puzzle, plus showed a few more challenges to be resolved!

So I’ll be adding a few posts on enhancing CC.NET to be distributed – either some of the issues involved or some of the pieces that have been added. Hopefully by the time we (finally) get to the 2.0 release CC.NET will be able to work as a distributed application :-)

That’s All For Now

So that’s what I’m planning on writing up over the next few weeks. At the same time I’ll be adding the source to SourceForge (under the CCNet2 branch) and hopefully spending some time with the other devs on getting CC.NET ready for the “official” 1.5 release.

Stay tuned…

Follow

Get every new post delivered to your Inbox.