1 comment

Lean systems as reincarnation of large ones


Washington SunglassesSome software systems are designed for massive amounts of data to be processed in a very short time. Banking systems, fraud detection, billing systems. Lets pick one, I worked on for a long time: billing systems (for telecom or internet providers for example).

Most of those systems are very large, mostly complex systems, designed to bill millions of customers per month. Some examples are Kenan Arbor (bought by Lucent) or Amdocs. Since these systems need to process vast amounts of data very fast, they are built using compiled sources / binaries. Binary software is not easily customizable. So most of these systems are widely customizable via configuration files. Taking in account all options possible in dynamic configurations results in even more compiled code. I think you got it.

What actually is a billing system?

To give an impression which steps are required in a typical billing scenario, here is a short non-exhaustive list:

  1. Preprocessing
    Collect billable items from external systems, translate or reformat the data and put it in a database.
    Additional for mobile telecom billing: import GSM TAP3 (Transferred Account Procedure version 3) roaming data.
  2. First step: Rating
    Put a “price tag” on every billable item for the billing period in question
  3. Second step: Billing
    Collect rated items as invoice items per customer.
  4. Third step: Invoicing
    Create invoices, on paper or digitally.
  5. Fourth step: Payment
    Withdraw money via saved payment option per customer.
    Alternatively: substract invoice total from prepaid deposit.

The problem

All of these steps could be “special” for any customers. Think of a subscribed service. Every customer pays $5 per month. But once the company had an introductory offer of 20% off for the first year. So not only are some customers paying only $4 but they are paying $5 after 12 months. Now take into account, that a typical mobile telecom company has something like 20-30 different contract types or rates. Wow, lots of options. Not a problem for a multi-million dollar company but for smaller companies with, let’s say, 100 to 100.000 customers.

A solution?

Now what if a billing system would be implemented in a scripting language? Admittedly it would be a bit slower (would it? I don’t really know) that a solution in C or C++. But it would be very fast and flexible customizable, if well documented (we developers love to write documentation, don’t we?). Also management summary dashboards would be much more flexible as prebuilt solutions like QlikView (which also would cost additional license fees).

I could visualize for example a solution in Python. This way it would be fast compared to some other scripting languages and could leverage the massive amount of financial and mathematical software components. Build an administration and dashboard component with Flask or Django and run Python scripts on a PostgreSQL database. If more speed is needed you could switch to an Apache Spark architecture, which would also be scriptable in Python via PySpark.

Start on a small budget but don’t let decisions limit your options!

0 comment

Basics of Trend Scouting, Part 1

This will be the start of a short series of postings dealing with trend scouting or trend forecasting. The goal is to get an overview of things happening in trend scouting so I can link to my main area of interest which is technology scouting.
We will get straight some basic vocabulary, have a look on how trends propagate through society and what the lifetime cycle of a trend looks like. Most of this basic knowledge will help us to understand how technology scouting and trends in technology work. So lets start with some basic terms.


A trend can be any sort of physical or emotional shift or movement in society. A trend is a movement in a defined direction. A style (which is often confused with trends) in contrast is where you arrive when you follow a trend. In consequence a trend is a line or way, a style is a point or location. A trend is a sociological anomaly or oddity being visible as sort of a peak in an otherwise rather flat cultural landscape.

Trend scout / Trend forecaster

Trend scouts and forecasters are people looking for the afore mentioned patterns or shifts in attitude, lifestyle or mindset which contradict common thinking or behaviour.

The spreading of trends

Some trends spread in analogy to an infection. The phenomenon of this type of “cultural gene” is often referred to as a meme, a term coined by Richard Dawkins in his book The Selfish Gene.
The mechanism of proliferation can be discribed by a theory of Everett Rogers called “Diffusion of Innovations”. Based on data by Bryce Ryan and Neal Gross who observed how farmers in Iowa adapted innovations in farming technology he identified several groups of people, who adopt innovative changes with different speed. There are several diffusion of innovation curve variants, but the following image shows a common one:
Innovators make up for 2.5% of all observed individuals, early adopters for around 13.5%. This shows that the fraction of people responsible for creating innovation and change is small. Its here where trend scouts and forecasters have to look for things coming.
In the next part we will have a look at how trends can be found and identified and how professional trend forecasters work.
To finish this first delivery I would like to list some literature references, which might be of interest:
Malcolm Gladwell, The Tipping Point: How Little Things Can Make a Big Difference
Martin Raymond, The Trend Forecaster’s Handbook