1 comment

Graph libraries – Cytoscape.js with Cola.js

Now it’s time for my favourite graph visualization framework. As an introduction I cite from the Cytoscape website:

Cytoscape is an open source software platform for visualizing molecular interaction networks and biological pathways and integrating these networks with annotations, gene expression profiles and other state data.

Cytoscape.js as a side project is a JavaScript library designed to build graph theory related data evaluation and visualization applications on the web. It comes with some layout renderers but is open to other visualization libraries such as Cola.js. Cola is a constraint based layout library with some very interesting features. In this context, we will use it only as a layout engine for the graph.

How does it look?

When playing around with this graph you will probably notice two things:

  1. When dragging around a single node all the other nodes of the graph remain calm and don’t wiggle around like in most physical based layouts.
  2. When reloading this page you will always see the same arrangement of nodes. The graph is stable to reloads.

These two feature result from the way Cola.js renders the graph and are very useful for real world applications. While it’s big fun wobbling and dragging around nodes in a real application you would always like to see the same graph when reloading the same data. And manipulating the arrangement should not affect nodes not touched. The complete source code can be found in the GitHub-Mark-32px GitHub repository.

How does it work?

Once again we start by including the stuff needed. And this are the two libraries and an adaptor to use Cola.js from within Cytoscape.js:

<script src="/graphs/common/cola.js"></script>
<script src="/graphs/common/cytoscape.js"></script>
<script src="/graphs/common/cytoscape-cola.js"></script>

The adaptor can also be found on GitHub.

The HTML body contains a div to render the graph into:

<div id="cy"></div>

The remainder of the index.html file consists of a script:

var elems = [
    {data: {id: '1', name: 'Node 1', nodecolor: '#88cc88'}},
    {data: {id: '2', name: 'Node 2', nodecolor: '#888888'}},
    {data: {id: '3', name: 'Node 3', nodecolor: '#888888'}},
    {data: {id: '4', name: 'Node 4', nodecolor: '#888888'}},
    {data: {id: '5', name: 'Node 5', nodecolor: '#888888'}},
    {data: {id: '6', name: 'Node 6', nodecolor: '#888888'}},
        data: {
            id: '12',
            source: '1',
            target: '2',
            linkcolor: '#888888'
        data: {
            id: '13',
            source: '1',
            target: '3',
            linkcolor: '#888888'
        data: {
            id: '24',
            source: '2',
            target: '4',
            linkcolor: '#ff8888'
        data: {
            id: '25',
            source: '2',
            target: '5',
            linkcolor: '#ff8888'
        data: {
            id: '45',
            source: '4',
            target: '5',
            linkcolor: '#ff8888'
        data: {
            id: '46',
            source: '4',
            target: '6',
            linkcolor: '#888888'
var cy = cytoscape({
    container: document.getElementById('cy'),
    elements: elems,
    style: cytoscape.stylesheet()
                'background-color': 'data(nodecolor)',
                label: 'data(name)',
                width: 25,
                height: 25
                'line-color': 'data(linkcolor)',
                width: 3
cy.layout({name: 'cola', padding: 10});

Lines 2 to 7 define the nodes with their name/caption and colour. Lines 8 to 54 define the edges in a similar way. Lines 56 to 70 define the Cytoscape container and attach dynamic properties to the nodes and edges. Line 71 finally kicks off the layout rendering. Note: up to that last line we specified exactly no rendering or geometry information! By changing the name of the renderer from cola  to one of the built-ins (circle  or cose  are fun to play with!) you can get a completely different outout.

As a last remark I strongly recommend to have a look at the graph theoretical and practical routines and algorithms the Cytoscape.js library has to offer.

0 comment

Graph libraries – Arbor.js

Arbor.js is written by Christian Swineheart from Samizdat Drafting Co. The documentation is concise even though the main page navigation as animated tree layout always drives me crazy :) The example I present here only depends on arbor.js and jQuery. Graph definition is for shortness and clarity done by calls to addNode  and addEdge  methods of the ParticleSystem renderer.

How does it look?

As you can see, the physical simulation is much more fluid and undamped than with Alchemy.js. I’m pretty much shure this can be changed to some degree with a different initialization of the particle system as described in the reference documentation. Once again, the complete source code, in this case consisting only of an index.html file, can be found in the GitHub-Mark-32px GitHub repository.

How does it work?

Once again in the header we start by loading the stuff needed:

<script src="/graphs/common/jquery-2.1.3.js"></script>
<script src="/graphs/common/arbor.js"></script>
<script src="/graphs/common/arbor-tween.js"></script>

arbor-tween.js is not really needed for this example as it allows gradual transitions in the data object of nodes and edges. Arbor.js by default renders its graph in a HTML5 canvas, in the body we need to define this:

<canvas id="viewport" width="800" height="500"></canvas>

The remainder of the file consists of a function defining and binding the renderer object to the canvas and constructing the graph:

(function ($) {
        var Renderer = function (canvas) {
                var canvas = $(canvas).get(0);
                var ctx = canvas.getContext("2d");
                var particleSystem;
                var that = {
                        init: function (system) {
                                particleSystem = system;
                                particleSystem.screenSize(canvas.width, canvas.height);
                        redraw: function () {
                                ctx.fillStyle = "white";
                                ctx.fillRect(0, 0, canvas.width, canvas.height);
                                particleSystem.eachEdge(function (edge, pt1, pt2) {
                                        ctx.strokeStyle =;
                                        ctx.lineWidth = 3;
                                        ctx.moveTo(pt1.x, pt1.y);
                                        ctx.lineTo(pt2.x, pt2.y);
                                particleSystem.eachNode(function (node, pt) {
                                        ctx.arc(pt.x, pt.y, 15, 0, 2 * Math.PI);
                                        ctx.fillStyle =;
                                        ctx.font = "18px Arial";
                                        ctx.fillStyle = "#000000";
                                        ctx.fillText(, pt.x + 20, pt.y + 5);
                        initMouseHandling: function () {
                                var dragged = null;
                                var handler = {
                                        clicked: function (e) {
                                                var pos = $(canvas).offset();
                                                _mouseP = arbor.Point(e.pageX - pos.left, e.pageY -;
                                                dragged = particleSystem.nearest(_mouseP);
                                                if (dragged && dragged.node !== null) {
                                                        dragged.node.fixed = true;
                                                $(canvas).bind('mousemove', handler.dragged);
                                                $(window).bind('mouseup', handler.dropped);
                                                return false;
                                        dragged: function (e) {
                                                var pos = $(canvas).offset();
                                                var s = arbor.Point(e.pageX - pos.left, e.pageY -;
                                                if (dragged && dragged.node !== null) {
                                                        var p = particleSystem.fromScreen(s);
                                                        dragged.node.p = p
                                                return false;
                                        dropped: function (e) {
                                                if (dragged === null || dragged.node === undefined) return;
                                                if (dragged.node !== null) dragged.node.fixed = false;
                                                dragged.node.tempMass = 1000;
                                                dragged = null;
                                                $(canvas).unbind('mousemove', handler.dragged);
                                                $(window).unbind('mouseup', handler.dropped);
                                                _mouseP = null;
                                                return false;
                return that;
        $(document).ready(function () {
                var sys = arbor.ParticleSystem(700, 700, 0.5);
                sys.renderer = Renderer("#viewport");
                sys.addNode('Node 1', {name: "Node 1", nodecolor: "#88cc88"});
                sys.addNode('Node 2', {name: "Node 2", nodecolor: "#888888"});
                sys.addNode('Node 3', {name: "Node 3", nodecolor: "#888888"});
                sys.addNode('Node 4', {name: "Node 4", nodecolor: "#888888"});
                sys.addNode('Node 5', {name: "Node 5", nodecolor: "#888888"});
                sys.addNode('Node 6', {name: "Node 6", nodecolor: "#888888"});
                sys.addEdge('Node 1', 'Node 3', {linkcolor: "#888888"});
                sys.addEdge('Node 1', 'Node 2', {linkcolor: "#888888"});
                sys.addEdge('Node 2', 'Node 5', {linkcolor: "#ff8888"});
                sys.addEdge('Node 2', 'Node 4', {linkcolor: "#ff8888"});
                sys.addEdge('Node 4', 'Node 5', {linkcolor: "#ff8888"});
                sys.addEdge('Node 5', 'Node 6', {linkcolor: "#888888"});

Lines 2 to 73 define the Renderer, in detail:

  • get the HTML5 canvas (line 3)
  • grab its 2D context (line 4)
  • an init function for the particle system (lines 7 to 12)
  • a redraw function actually drawing the graph (lines 13 to 33) by iterating over all nodes and edges
  • defining a pretty basic mouse handling (lines 34 to 69)

In line 75 we get a particle system, set the rendering to the canvas in line 76 and define nodes (lines 76 to 81) and edges (lines 82 to 87). That’s pretty much all.


Graph libraries – Alchemy.js

Alchemy.js is a library developed by graphAlchemist (site defunct). The documentation could be better and more detailed but it works. Like many other libraries it leverages other libraries to do “the basic stuff”. In this case it is it depends on:

Alchemy uses GraphJSON (similar to GeoJSON) as input format, which is quite nice but undocumented since the website doesn’t exist any more. Looks as if support for it ceded with the graphAlchemist web site …

How does it look?

Like all demos I’ll show it’s a fully functional mini demo, so you can drag the nodes around. I’ll present excerpts of the code to explain. The complete code can be found in a GitHub-Mark-32px GitHub repository.

How does it work?

Our example consists of an index.html file, a GraphJSON data file and the library stuff. In the index.html we first load the stuff needed:

<link rel="stylesheet" href="/graphs/common/alchemy/alchemy.css"/>
<script src="/graphs/common/jquery-2.1.3.js"></script>
<script src="/graphs/common/d3.js"></script>
<script src="/graphs/common/lodash.js"></script>

The body contains a div to put the graph in and the include for the base library itself:

<div class="alchemy" id="alchemy"></div>
<script src="/graphs/common/alchemy/alchemy.js"></script>

Finally there’s the code producing the graph:

var config = {
    dataSource: '/graphs/alchemy/data/network.json',
    graphWidth: function() {return 800},
    graphHeight: function() {return 500},
    backgroundColor: "#ffffff",
    nodeCaptionsOnByDefault: true,
    caption: function(node){
        return node.caption;
    nodeTypes: {
        "role": ["greyone", "greenone"]
    edgeTypes: {
        "caption": ["greyone", "redone"]
    nodeStyle: {
        "greyone": {
            color: "#888888",
            borderWidth: 0,
            radius: 15
        "greenone": {
            color: "#88cc88",
            borderWidth: 0,
            radius: 15
    edgeStyle: {
        "greyone": {
            color: "#888888",
            width: 3
        "redone": {
            color: "#ff8888",
            width: 3
alchemy = new Alchemy(config)

As you can see, nearly everything is done in a configuration object. Line 2 defines the source of the GraphJSON file, lines 3 & 4 set the dimensions of the SVG element. The setting of nodeCaptionsOnByDefault  in line 6 switches on the display of the node titles. Otherwise they would only show up when hovering with the mouse. Lines 7 to 9 define a callback function defining what to display as a caption / title. Lines 10 to 15 define valid values for grouping edges and nodes. Nodes are grouped by the role attribute, edges by caption. Lines 16 to 37 define how these groups of edges and nodes should be formatted. There are other, more direct ways to style a graph in Alchemy.js but I wanted to show the (for me) most elegant way. Finally line 39 kicks off the rendering of the graph with the given configuration.

0 comment

Graph libraries – Introduction

This post starts a small series of articles, that will present some JavaScript graph plotting libraries and show how to draw more or less the same simple demo graph with these different libs.

What is a graph?

Graph theory is a mathematical discipline that studies networks of nodes and links or edges connecting them. Graphs are useful for many applications: social media networks, streets connecting cities, transfers connecting bank accounts or networks of people, addresses, bank accounts and cars connected via insurance claims and events. We’ll deal with some of those applications later on when discussing the application of network and graph paradigms to visualizing fraudulent actions.

Which libraries have I examined?

The internet is full, I mean literally FULL of libraries to display networked graphs. I had to choose some and discard others. First I decided to neglect commercial libraries that don’t have an open source edition. Then I stopped evaluating libs when I recognized that the documentation is too poor or functionality too limited to further investigate. So here is the list in alphabetical order:

Libraries I didn’t evaluate due to licensing constraints:

Libraries I gave up on:

  • Dracula Graph Library (Graph Dracula … you understand … haha)
    Needs a raphael.js renderer to display custom node shapes. Too much fuzz for a very short test.
  • sigma.js
    Poorly documented. Didn’t succeed to build a test candidate in short (!) time.

How did I test?

networkI always used the same test network. It consists of 6 nodes and 6 edges. Some edges are red and one node is green. The nodes are labelled “Node 1” to “Node 6”. Colouring and labelling is used to test how easy it is to insert some basic customization into the layout.

On the right side you can see a hand drawn (well not really handdrawn) picture of the demo network. The examples I present are fully functional, not screen shots and I will briefly go through the source code.


The first part for Alchemy.js can be found here.

The second post on Arbor.js is here.

The third part with Cytoscape.js and Cola.js is here.

The fourth part with D3 is here.

1 comment

Lean systems as reincarnation of large ones


Washington SunglassesSome software systems are designed for massive amounts of data to be processed in a very short time. Banking systems, fraud detection, billing systems. Lets pick one, I worked on for a long time: billing systems (for telecom or internet providers for example).

Most of those systems are very large, mostly complex systems, designed to bill millions of customers per month. Some examples are Kenan Arbor (bought by Lucent) or Amdocs. Since these systems need to process vast amounts of data very fast, they are built using compiled sources / binaries. Binary software is not easily customizable. So most of these systems are widely customizable via configuration files. Taking in account all options possible in dynamic configurations results in even more compiled code. I think you got it.

What actually is a billing system?

To give an impression which steps are required in a typical billing scenario, here is a short non-exhaustive list:

  1. Preprocessing
    Collect billable items from external systems, translate or reformat the data and put it in a database.
    Additional for mobile telecom billing: import GSM TAP3 (Transferred Account Procedure version 3) roaming data.
  2. First step: Rating
    Put a “price tag” on every billable item for the billing period in question
  3. Second step: Billing
    Collect rated items as invoice items per customer.
  4. Third step: Invoicing
    Create invoices, on paper or digitally.
  5. Fourth step: Payment
    Withdraw money via saved payment option per customer.
    Alternatively: substract invoice total from prepaid deposit.

The problem

All of these steps could be “special” for any customers. Think of a subscribed service. Every customer pays $5 per month. But once the company had an introductory offer of 20% off for the first year. So not only are some customers paying only $4 but they are paying $5 after 12 months. Now take into account, that a typical mobile telecom company has something like 20-30 different contract types or rates. Wow, lots of options. Not a problem for a multi-million dollar company but for smaller companies with, let’s say, 100 to 100.000 customers.

A solution?

Now what if a billing system would be implemented in a scripting language? Admittedly it would be a bit slower (would it? I don’t really know) that a solution in C or C++. But it would be very fast and flexible customizable, if well documented (we developers love to write documentation, don’t we?). Also management summary dashboards would be much more flexible as prebuilt solutions like QlikView (which also would cost additional license fees).

I could visualize for example a solution in Python. This way it would be fast compared to some other scripting languages and could leverage the massive amount of financial and mathematical software components. Build an administration and dashboard component with Flask or Django and run Python scripts on a PostgreSQL database. If more speed is needed you could switch to an Apache Spark architecture, which would also be scriptable in Python via PySpark.

Start on a small budget but don’t let decisions limit your options!

0 comment

Changing filenames to camel case

Sometimes you’ll encounter the task to rename files named with snake_case to CamelCase. And sometimes there are a lot of those files. For example when porting a CakePHP 1 project to CakePHP 2 or 3. In CakePHP the upgrade console does a decent job renaming a lot of files for you. But in larger projects having subfolders you’re left with an awful lot of unrenamed scripts. This is where my Python 3 script comes in. It renames any filenames in the current directory given as parameters (thanks to Python 3 argparse you can use wildcards!) from snake_case to CamelCase filename. It preserves the extension if the filename has one. It also has a quiet mode (use -q) to suppress any output at the command line and a preview mode just like GNU make (use -n, supersedes -q), which doesn’t do anything but print out what it would have done.

(Picture by Startup Stock Photos CC 0,

0 comment

Knowledge series, Part 1 – Human capital vs. Human resources

2631466945_de1bbc2cfd_zThe term human capital was coined by american economist Theodore Schultz in his seminal paper “Investment in Human Capital” from 1961 [1]. He starts by discussing that often the consideration of human beings as a form of capital is seen as unethical. To count human beings as capital is justified by a rationale drawing on comparisons and discussions from the beginning of the 20th century, comparing e.g. this approach to sacrificing a hundred human beings in order to save a gun in war. Well, our views on war have changed since 1906. This might explain why today we tend to abhor counting humans as capital.

But there is a second approach originating in the theories of french sociologist Pierre Bourdieu readily summarized in 1986 [2]. He derives his concept of cultural capital from observations of french upper class children gaining substantial advantages in their life compared to working class children resulting from their education. He states that their parents invest economic capital (money) in the education of their children. They accumulate a certain amount of cultural capital which can later on be converted back to economic capital when the owners of the cultural capital get more profitable jobs because of their better education. This cultural capital is saved in what he calls the embodied form which means that the knowledge and education belongs naturally to the person acquiring it. It can not be sold like a physical item. When investing e.g. in a collection of paintings this capital can be seen as transformed into another form of economic capital since the paintings have a certain economic value. But the consumption and appreciation of art and its display at social events also counts as a form of cultural capital, this time in objectified state. When acquiring an academic title the title itself represents the embodied cultural capital of the title holder in a certain institutionalized state.

In contrast the term human resources is nowadays widely used to describe all that concerns staffing or personnel management. It was first used by american economist John Commons in 1893 [3] and was commonplace (please excuse the pun) in economic literature from the 1910s on. It represents the view of employees as assets to the firm. This interpretation is opposed by the preamble of the United Nations International Labour Organization including the principle “Labour is not a commodity“.

When forced to  choose one of the two terms I would opt for “human capital” since its philosophical and sociological implications put the human being at least a bit more in the focus of interest.


[1] Schultz, T. W.. (1961). Investment in Human Capital. The American Economic Review, 51(1), 1–17

[2] The Forms of Capital: English version published 1986 in J.G. Richardson’s Handbook for Theory and Research for the Sociology of Education, pp. 241–258.

[3] John R. Commons, The Distribution of Wealth, New York, 1893

(Picture by Ian Muttoo CC BY-ND 2.0,

0 comment

Knowledge in and out

wp-1456109997332.jpgIn my last post I mused about employer policies concerning training and learning of employees. I would like to use this as an introduction to a series of postings related to knowledge management and innovation in firms. Much has been written and discussed about the creation and finding of knowledge. From a practical point of view and in philosophy, which discusses these topics as epistemology. In this first installment I would like to give an overview of what’s ahead and how I would like to structure my texts.

My first approach was to put all of this in two or three postings. Since this is way too much text to be read comfortably I will divide the topics in a more or less natural way like this:

  1. The idea of human capital in contrast to human resources
  2. Philosophy and theory of knowledge, from information via knowledge to innovation
  3. Flow of knowledge in the firm and exchange with external resources
  4. Knowledge management from a social network point of view
  5. Challenges for human resources departments concerning knowledge management
  6. Impediments for knowledge creation and usage (e.g. Not-Invented-Here-Syndrome, NIH)
  7. The state of open innovation
  8. Simulation of knowledge flow and NIH using cellular automata

What do you think? Interested? Is something missing? Should I reorder the topics? Let me know!

(Picture by Tommy Ellis CC BY-ND 2.0,

1 comment

Degradation on the job

I remember a conversation with my father-in-law. We were discussing the transition of my freelancing work into an employment. We spoke about the salary and benefits in the contract. He asked, if I had included a surcharge for being on a permanent task. I asked him to explain. And here is the bottom line.

Photo from Death to Stock photos

We (in this industry) are knowledge workers. This is a much overly used phrase. It boils down to that our value as an employee or freelancer depends on our knowledge, be it explicit or tacit. If we are hired for a certain position, we need to keep at it. There are several possibilities how we can do that. Going to conferences, taking a training, meet colleagues etc. This needs time. And sometimes money. All this expenditure has to be accounted for. By you or by your employer. He may support you with money or free time or he may not (which is common in Germany). If not the salary has to compensate for your personal investment. So why has he to do that?

Because his order to work on a certain task or field for a (supposedly) very long time distracts you from acquiring additional knowledge. If you would still be freelancing, hopping from gig to gig you would to some extent learn new things. And you would invest some of your time and / or money in getting new things to know. Since you are your own boss.

If you work on a defined focussed task for a longer time, you will be left behind on your fields of expertise. This once was the reason you were hired. After some years, your worth as a coworker will degrade. And this is the reason good employers care for your knowledge.

1 comment

How to merge two WordPress blogs

Introduction and Overview

For some of you it might be pretty obvious how to do that. But for some of my readers this might be a new topic. So here is a short overview of how to do it:

  1. Test if it works:
    1. Prequisite: you’ll need a locally running web server with PHP support. If you don’t have one, you could try to test the merge on a second web server, but alas that’s out of the scope of this posting.
    2. Download the destination blog and install it locally.
    3. Install the WordPress Command Line Tool.
    4. Download all articles from the source blog as XML file.
    5. Install the WordPress Import plugin.
    6. Try to import the XML file.
  2. If all works well, repeat that (without step 3 since the URLs are ok for production) in the production instance of the destination blog.

Local test

Download destination blog

First we need a download of the blog, where all postings should be merged together. This means:

  • Zip or tar the whole WordPress directory together, get it down to your machine.
  • Install it as a virtual host on your local web server.
  • Get a full export of your WordPress database.

My WordPress directory was called “www” and resided in /opt/sites/de/technologyscout (that’s how all my web server document roots  are organized /opt/sites/TLD/DOMAIN/SUBDOMAIN). So I downloaded it to my laptop, moved it to /opt/sites (yes, I maintain that also locally) and renamed it to “techscout”.

If you’re like me, it’s a good idea, to look up the database credentials in WORDPRESSROOT/wp-config.php:

define('DB_NAME', 'xxxxx');

define('DB_USER', 'yyyyy');

define('DB_PASSWORD', 'zzzzz');

I also like to have exactly the same credetials locally, so at the MySQL command line I do:

grant all on xxxxx.* to yyyyy@localhost identified by 'zzzzz';

flush privileges;

Then I import the downloaded WordPress database with:

mysql -uyyyyy -pzzzzz xxxxx < database.sql


Install WordPress Command Line Tool

Since WordPress has a habit of saving absolute URLs in the database, which is btw. a very bad and stupid habit, we need to patch those URLs to work locally. This is done with the indispensable wp-cli tool:

wp search-replace '' 'http://techscout'

This will replace every occurrence of with http://techscout. Now please try to log in with your credentials. Everything should work locally now. Please have a look at the URL in your browser after login. If you see the productive URL (for me this means the search and replace didn’t work as expected.

Download XML archive with all postings

Please go to Tools -> Export in the admin and select “Posts”. You’ll see additional options. I selected:

  • Categories: All
  • Authors: All
  • Start date, End date: Nothing, just go with –Select–
  • Status: All (You can only import published posts, if you like)

Now click on “Download Export File” and save the file locally.

Install WordPress Import plugin

Now please go to Tools -> Import inside the admin panel. You will see a list of options similar to this one:


Please select the last entry called “WordPress” (who could guess that, right?). If you didn’t have the WordPress import plugin installed WordPress will ask you if you would like to do so right now. We certainly want to. After installing the plugin, we are ready to …

Import the WordPress XML file

Proceed by selecting the input file we just downloaded. Everything should work smoothly. The imported postings should be sorted according to their original date into the database.

Do it in production

It’s a good idea to get a database backup of your production destination database before importing the blog posts. Then go to Tools -> Import, install the import plugin if needed, and import the XML file we already used locally.

That’s all. Have fun!