Setting up McCabe Code Complexity reporting for Python in Jenkins (Hudson)

In this post I’m going to explain the steps in order to integrate pycabehtml https://bitbucket.org/jordilin/pycabehtml  into Jenkins using the Html Plugin.

1. Add build step Execute Shell.
2. You’ll see a text box where you will be able to write the shell script that will be executed in order to build your project and obtain metrics.
3. Write the following in the text box.

cd $WORKSPACE
COMPLEXITY_DIR=$WORKSPACE/complexity
GRAPH=$COMPLEXITY_DIR/output.png
ACC=/usr/local/jenkins/jenkinsprojectname/complexity_acc.txt
mkdir -p $COMPLEXITY_DIR
pymetrics `find yoursrc/ -iname "*.py"` > $COMPLEXITY_DIR/complexity.txt
pycabehtml.py -i $COMPLEXITY_DIR/complexty.txt -o $COMPLEXITY_DIR/metrics.html -a $ACC -g $GRAPH

4. You can save this script into a shell script and put it under source control (recommended), and then execute this shell script from the jenkins build step execute shell.

5. Enable the Html plugin for Jenkins

Reporting McCabe Code Complexity for Python projects in Jenkins (aka Hudson)

Not long ago, I wrote a parser for Pymetrics to produce an html formatted document designed to be integrated in Jenkins using its Html plugin. In Jenkins CI we have Python support for unit tests, code coverage and lint reporting, but not for code complexity as far as I know.

Pycabehtml tries to solve this problem by parsing the output of Pymetrics and report the trends on SLOC (source lines of code), Number of Python modules, and McCabe code complexity over builds. On top of that, it showcases all those modules and methods which have McCabe index over 10 in temperature like colors. Methods with index 10 to 20 are highlighted in yellow, 20 to 50 in orange and over 50 in red.

In Python, due to its dynamic nature, no code should be beyond 10. If any method does, then its time for a refactor. The sooner you catch those, the better; or it might be too late.

Pycabehtml is licensed GPLv3 and is hosted in bitbucket at: https://bitbucket.org/jordilin/pycabehtml/overview

Here I leave attached just a couple of portions of the report done on the core Django web application framework.

In Jenkins/Hudson would appear a link to the above report (you need to install the Html plugin). The next screenshot is for the opensource project alerta that can be found at: https://bitbucket.org/jordilin/alerta/wiki/Home

Deploying Django applications in a scalable way

Pip or easy_install inside a virtual environment is great for development
purposes when you want all your dependencies isolated from your system. It’s
great to manage dependencies and reproduce the same development environment
across different developers at any given time. But, it has a great con. It’s
not portable and not scalable. If you want to deploy virtualenvs in hundreds of
servers you get a big problem when you need to recreate the environment in
every single server and pull the code. Sure, you can automate that, but still
it is an error prone and slow way to do it, as you will need to hit the net
(internal or external) in order to reproduce the environment.

The recommended way to do it is using Buildout and the target operating
system’s native package management system. If your production OS is Debian or
derivatives (Ubuntu) then you should package your app in a DEB package. If you
use RedHat or derivatives, then you should package your app in an RPM package.
You will have the extreme power of apt-get or yum. Freezing (holding) and
rollbacks are completely integrated. You can deploy in parallel to hundreds of
servers in an incredible fast manner. And fast rollbacking if you get some sort
of problem…

Pip and easy_install in a virtualenv can still be used along with buildout.
But, your CI server should take buildout as a preference taking all the pip or
easy_install dependencies as package dependencies. You can make apt-get and yum
deal with pip dependencies and inject all the buildout eggs in your deb package
in a portable way.

You can take a look at an excellent tutorial on how to use buildout at  http://jacobian.org/writing/django-apps-with-buildout/ by Jacob Kaplan-Moss.

log4tailer under the Alerta Project in bitbucket

Log4tailer project has been moved to bitbucket under the Alerta Project http://bitbucket.org/jordilin/alerta. The main objective is to provide a complete log monitoring application.

Alsa 1.0.23 in Debian Squeeze

Debian squeeze comes with Alsa drivers 1.0.21 by default:

cat /proc/asound/version

and you’ll see which version you have installed. Unfortunately my Sony Vaio needs Alsa 1.0.23 to work. Debian Squeeze has alsa-source for 1.0.23 in the repository, so it’s only a question to go into a terminal and use module assistant.

m-a a-i alsa

an ncurses screen will show up while compiling the sources. Just restart and verify that you have the new version.

Log4tailer in its version 2.9 released

Log4tailer 2.9 has been released and it provides a couple of new features that will make monitoring your application logs even better: PrintShot and Poster.

• PrintShot notification just takes an screenshot of your screen whenever an alertable log trace is found. I find this feature quite cool, mostly whenever I’m writing documentation. It can be interesting when you want some proof of a trace that has been logged as well.
• Poster notification sends requests to a web server that can store logtraces. This will provide the foundation for Log4server, a centralized web server that will receive notifications from log4tailers spread across a network. Log4server is under development at http://code.google.com/p/log4tailer/

Log4tailer 2.9 can be downloaded from its website at http://code.google.com/p/log4tailer/

Ubuntu Maverick not a perfect 10, but nearly.

I say that because I’ve found at least a couple of issues that I didn’t find in Lucid. First one is the Intel IPS thermal kernel messages that don’t stop shouting at /var/log/messages. Traces like “MCP power or thermal limit exceeded” are showing up every 2 seconds, which is kind of really annoying to say the least. That module was activated (as far as I’ve investigated) in the Maverick release, so in order to avoid you logs growing fast, you can disable that by doing the following:

• Edit file /etc/modprobe.d/blacklist.conf
• Add at the end of the file blacklist intel_ips
• Save file and restart.

After restart, if you tail /var/log/messages you should not see that log trace again.

The second issue that I’ve found, and I haven’t been able to solve is the wifi. I used to have a propietary driver Broadcom STA available to be installed from the same Ubuntu, but it seems that it has been removed. I haven’t had time yet to investigate that, so hopefully we will find some workaround soon.