Article
1 comment

Lean systems as reincarnation of large ones

Introduction

Washington SunglassesSome software systems are designed for massive amounts of data to be processed in a very short time. Banking systems, fraud detection, billing systems. Lets pick one, I worked on for a long time: billing systems (for telecom or internet providers for example).

Most of those systems are very large, mostly complex systems, designed to bill millions of customers per month. Some examples are Kenan Arbor (bought by Lucent) or Amdocs. Since these systems need to process vast amounts of data very fast, they are built using compiled sources / binaries. Binary software is not easily customizable. So most of these systems are widely customizable via configuration files. Taking in account all options possible in dynamic configurations results in even more compiled code. I think you got it.

What actually is a billing system?

To give an impression which steps are required in a typical billing scenario, here is a short non-exhaustive list:

  1. Preprocessing
    Collect billable items from external systems, translate or reformat the data and put it in a database.
    Additional for mobile telecom billing: import GSM TAP3 (Transferred Account Procedure version 3) roaming data.
  2. First step: Rating
    Put a “price tag” on every billable item for the billing period in question
  3. Second step: Billing
    Collect rated items as invoice items per customer.
  4. Third step: Invoicing
    Create invoices, on paper or digitally.
  5. Fourth step: Payment
    Withdraw money via saved payment option per customer.
    Alternatively: substract invoice total from prepaid deposit.

The problem

All of these steps could be “special” for any customers. Think of a subscribed service. Every customer pays $5 per month. But once the company had an introductory offer of 20% off for the first year. So not only are some customers paying only $4 but they are paying $5 after 12 months. Now take into account, that a typical mobile telecom company has something like 20-30 different contract types or rates. Wow, lots of options. Not a problem for a multi-million dollar company but for smaller companies with, let’s say, 100 to 100.000 customers.

A solution?

Now what if a billing system would be implemented in a scripting language? Admittedly it would be a bit slower (would it? I don’t really know) that a solution in C or C++. But it would be very fast and flexible customizable, if well documented (we developers love to write documentation, don’t we?). Also management summary dashboards would be much more flexible as prebuilt solutions like QlikView (which also would cost additional license fees).

I could visualize for example a solution in Python. This way it would be fast compared to some other scripting languages and could leverage the massive amount of financial and mathematical software components. Build an administration and dashboard component with Flask or Django and run Python scripts on a PostgreSQL database. If more speed is needed you could switch to an Apache Spark architecture, which would also be scriptable in Python via PySpark.

Start on a small budget but don’t let decisions limit your options!

Article
0 comment

Assholes and code of conduct manifests

Recently someone pointed out, that the perceived number of assholes is rising. This means, that anti-social behavior can be encountered more often. I responded that there seems to be some sort of sociological mechanism leading to anti-social behavior being more accepted than before.
Someone else asked what that mechanism might be, so here is my amateurish point of view.
Social interaction often is accomplished by communication. So examining forms of anti-social or disruptive communicational behavior might help to clarify some points. One form of disruptive communication is the so called “interactive vandalism” (Anthony Giddens, Sociology). Giddens points out, that effective communication or interaction is based on a cooperative behavior of the participants. If one party of an interaction deliberately behaves in a non-cooperative way, this often is encountered as a aggressive attitude by the other participants. But this is a stylistic device, not an explanation.
Another perspective is that of Erving Goffman‘s “The Presentation of Self in Everyday Life“: people behave as if they were acting. And like in a theater there is a stage and a backstage area. In the front region (stage) they act mostly according to common sense rules. In the back region (backstage) they can “give vent to feelings and styles of behavior they keep in check when on stage”. So acting more anti-social might mean transferring behavioral patterns from back to front.
In traditional social settings this would in general have been a completely unacceptable behavior, but why is it not judged that way now? There is a general process at work transferring the private into the public. Reality shows on TV, social media, liberalization of professional situations. Don’t get me wrong: I don’t judge those processes, but if not backed by a so called “good education” things can go wrong unnoticed. If this happens and some sort of “invisible control” doesn’t come into effect, openly visible regulation is a way to prevent unwanted situations.
An area where this currently happens are public conferences. More and more conference hosts issue code of conduct manifests. I got into some serious discussions because I objected that these rules are pretty obvious and stating them so explicitly might be sort of embarrassing for “well-behaved visitors”. Conference organizers assured me that these rules are not so obvious anymore. Maybe in the future we’ll see explicitly stated rules for human interaction more often. While being liberated from unnecessarily rigid forms of social behavior is a good thing, this feels like a cultural loss to me. But than again I just might be getting old.

Article
1 comment

Science and technology links [22-05-2016]

2652953996_76184c9ef4_z

Here are some interesting links connecting technology and science.

Open Scholar

Open Scholar is a open source content management system based on Drupal. This means it’s written in PHP. It is developed at Harvard University at the Institute of Quantitative Social Science.

Open Science Framework

OSF is a management system for scientific projects. It can be used as a hosted service right at the site or self-hosted. OSF is witten in Python and developed by the Center for Open Science.

The Dataverse Project

The Dataverse Project is a research data repository software. There are at the time of this writing 17 interconnected installations around the world. The software is written in Java and is developed by the aforementioned Institute of Quantitative Social Science at Harvard.

Figshare

Figshare is a repository for research output (mainly publications or papers). It’s a closed-source hosted service developed by Digital Science which also developed Overleaf. More on scientific writing solutions in a later post.

Article
0 comment

How to read in a statistic whatever you want

Professor at chalkboardIt’s a common place that you can read nearly any result you want from a statistic. You just have to optimize your mathematical model or cut short the reasoning about the data. There is currently a JAMA publication from the American Medical Society which is cited in many magazines and newspapers (even in German Spiegel) as “22% less risk of colorectal cancer for vegetarians”. No ordinary reader of these reviews will have a look at the original numbers in the publication since it is not freely available (yet another reason for open Publication …). But here they are:

  • Vegetarian participants: 40367
    Cancer cases: 252
  • Nonvegetarian participants: 37292
    Cancer cases: 238

This makes for the following relative case numbers:

  • Vegetarian: 0,624 / 100 participants
  • Nonvegetarian: 0,638 / 100 participants

Or a difference of 0,014 cases per 100 people. This means, if you eat meat your risk to come down with a form of colorectal cancer increases by 0,014 percent. This reads quite different, doesn’t it?

Article
0 comment

Changing filenames to camel case

Sometimes you’ll encounter the task to rename files named with snake_case to CamelCase. And sometimes there are a lot of those files. For example when porting a CakePHP 1 project to CakePHP 2 or 3. In CakePHP the upgrade console does a decent job renaming a lot of files for you. But in larger projects having subfolders you’re left with an awful lot of unrenamed scripts. This is where my Python 3 script comes in. It renames any filenames in the current directory given as parameters (thanks to Python 3 argparse you can use wildcards!) from snake_case to CamelCase filename. It preserves the extension if the filename has one. It also has a quiet mode (use -q) to suppress any output at the command line and a preview mode just like GNU make (use -n, supersedes -q), which doesn’t do anything but print out what it would have done.

(Picture by Startup Stock Photos CC 0, http://startupstockphotos.com/)

Article
0 comment

Knowledge series, Part 1 – Human capital vs. Human resources

2631466945_de1bbc2cfd_zThe term human capital was coined by american economist Theodore Schultz in his seminal paper “Investment in Human Capital” from 1961 [1]. He starts by discussing that often the consideration of human beings as a form of capital is seen as unethical. To count human beings as capital is justified by a rationale drawing on comparisons and discussions from the beginning of the 20th century, comparing e.g. this approach to sacrificing a hundred human beings in order to save a gun in war. Well, our views on war have changed since 1906. This might explain why today we tend to abhor counting humans as capital.

But there is a second approach originating in the theories of french sociologist Pierre Bourdieu readily summarized in 1986 [2]. He derives his concept of cultural capital from observations of french upper class children gaining substantial advantages in their life compared to working class children resulting from their education. He states that their parents invest economic capital (money) in the education of their children. They accumulate a certain amount of cultural capital which can later on be converted back to economic capital when the owners of the cultural capital get more profitable jobs because of their better education. This cultural capital is saved in what he calls the embodied form which means that the knowledge and education belongs naturally to the person acquiring it. It can not be sold like a physical item. When investing e.g. in a collection of paintings this capital can be seen as transformed into another form of economic capital since the paintings have a certain economic value. But the consumption and appreciation of art and its display at social events also counts as a form of cultural capital, this time in objectified state. When acquiring an academic title the title itself represents the embodied cultural capital of the title holder in a certain institutionalized state.

In contrast the term human resources is nowadays widely used to describe all that concerns staffing or personnel management. It was first used by american economist John Commons in 1893 [3] and was commonplace (please excuse the pun) in economic literature from the 1910s on. It represents the view of employees as assets to the firm. This interpretation is opposed by the preamble of the United Nations International Labour Organization including the principle “Labour is not a commodity“.

When forced to  choose one of the two terms I would opt for “human capital” since its philosophical and sociological implications put the human being at least a bit more in the focus of interest.

References

[1] Schultz, T. W.. (1961). Investment in Human Capital. The American Economic Review, 51(1), 1–17

[2] The Forms of Capital: English version published 1986 in J.G. Richardson’s Handbook for Theory and Research for the Sociology of Education, pp. 241–258.

[3] John R. Commons, The Distribution of Wealth, New York, 1893

(Picture by Ian Muttoo CC BY-ND 2.0, https://www.flickr.com/photos/imuttoo/2631466945/)

Article
0 comment

Knowledge in and out

wp-1456109997332.jpgIn my last post I mused about employer policies concerning training and learning of employees. I would like to use this as an introduction to a series of postings related to knowledge management and innovation in firms. Much has been written and discussed about the creation and finding of knowledge. From a practical point of view and in philosophy, which discusses these topics as epistemology. In this first installment I would like to give an overview of what’s ahead and how I would like to structure my texts.

My first approach was to put all of this in two or three postings. Since this is way too much text to be read comfortably I will divide the topics in a more or less natural way like this:

  1. The idea of human capital in contrast to human resources
  2. Philosophy and theory of knowledge, from information via knowledge to innovation
  3. Flow of knowledge in the firm and exchange with external resources
  4. Knowledge management from a social network point of view
  5. Challenges for human resources departments concerning knowledge management
  6. Impediments for knowledge creation and usage (e.g. Not-Invented-Here-Syndrome, NIH)
  7. The state of open innovation
  8. Simulation of knowledge flow and NIH using cellular automata

What do you think? Interested? Is something missing? Should I reorder the topics? Let me know!

(Picture by Tommy Ellis CC BY-ND 2.0, https://www.flickr.com/photos/tommyellis/6326877778/)

Article
1 comment

Degradation on the job

I remember a conversation with my father-in-law. We were discussing the transition of my freelancing work into an employment. We spoke about the salary and benefits in the contract. He asked, if I had included a surcharge for being on a permanent task. I asked him to explain. And here is the bottom line.

Photo from Death to Stock photos http://deathtothestockphoto.com/

We (in this industry) are knowledge workers. This is a much overly used phrase. It boils down to that our value as an employee or freelancer depends on our knowledge, be it explicit or tacit. If we are hired for a certain position, we need to keep at it. There are several possibilities how we can do that. Going to conferences, taking a training, meet colleagues etc. This needs time. And sometimes money. All this expenditure has to be accounted for. By you or by your employer. He may support you with money or free time or he may not (which is common in Germany). If not the salary has to compensate for your personal investment. So why has he to do that?

Because his order to work on a certain task or field for a (supposedly) very long time distracts you from acquiring additional knowledge. If you would still be freelancing, hopping from gig to gig you would to some extent learn new things. And you would invest some of your time and / or money in getting new things to know. Since you are your own boss.

If you work on a defined focussed task for a longer time, you will be left behind on your fields of expertise. This once was the reason you were hired. After some years, your worth as a coworker will degrade. And this is the reason good employers care for your knowledge.

Article
1 comment

How to merge two WordPress blogs

Introduction and Overview

For some of you it might be pretty obvious how to do that. But for some of my readers this might be a new topic. So here is a short overview of how to do it:

  1. Test if it works:
    1. Prequisite: you’ll need a locally running web server with PHP support. If you don’t have one, you could try to test the merge on a second web server, but alas that’s out of the scope of this posting.
    2. Download the destination blog and install it locally.
    3. Install the WordPress Command Line Tool.
    4. Download all articles from the source blog as XML file.
    5. Install the WordPress Import plugin.
    6. Try to import the XML file.
  2. If all works well, repeat that (without step 3 since the URLs are ok for production) in the production instance of the destination blog.

Local test

Download destination blog

First we need a download of the blog, where all postings should be merged together. This means:

  • Zip or tar the whole WordPress directory together, get it down to your machine.
  • Install it as a virtual host on your local web server.
  • Get a full export of your WordPress database.

My WordPress directory was called “www” and resided in /opt/sites/de/technologyscout (that’s how all my web server document roots  are organized /opt/sites/TLD/DOMAIN/SUBDOMAIN). So I downloaded it to my laptop, moved it to /opt/sites (yes, I maintain that also locally) and renamed it to “techscout”.

If you’re like me, it’s a good idea, to look up the database credentials in WORDPRESSROOT/wp-config.php:

I also like to have exactly the same credetials locally, so at the MySQL command line I do:

Then I import the downloaded WordPress database with:

 

Install WordPress Command Line Tool

Since WordPress has a habit of saving absolute URLs in the database, which is btw. a very bad and stupid habit, we need to patch those URLs to work locally. This is done with the indispensable wp-cli tool:

This will replace every occurrence of http://www.technologyscout.de with http://techscout. Now please try to log in with your credentials. Everything should work locally now. Please have a look at the URL in your browser after login. If you see the productive URL (for me http://www.technologyscout.de) this means the search and replace didn’t work as expected.

Download XML archive with all postings

Please go to Tools -> Export in the admin and select “Posts”. You’ll see additional options. I selected:

  • Categories: All
  • Authors: All
  • Start date, End date: Nothing, just go with –Select–
  • Status: All (You can only import published posts, if you like)

Now click on “Download Export File” and save the file locally.

Install WordPress Import plugin

Now please go to Tools -> Import inside the admin panel. You will see a list of options similar to this one:

Importoptions

Please select the last entry called “WordPress” (who could guess that, right?). If you didn’t have the WordPress import plugin installed WordPress will ask you if you would like to do so right now. We certainly want to. After installing the plugin, we are ready to …

Import the WordPress XML file

Proceed by selecting the input file we just downloaded. Everything should work smoothly. The imported postings should be sorted according to their original date into the database.

Do it in production

It’s a good idea to get a database backup of your production destination database before importing the blog posts. Then go to Tools -> Import, install the import plugin if needed, and import the XML file we already used locally.

That’s all. Have fun!

Article
0 comment

Blogumzug – Blog move

Deutsch

Hallo, vielleicht wundern Sie sich, daß Sie auf dieser Seite landen, obwohl Sie die deutsche Version sehen wollten. Ich habe festgestellt, daß es auf Dauer eher unsinnig ist, zwei Blogs mit ähnlichen Inhalten zu verwalten und habe daher die beiden Blogs zusammen gelegt. Die Inhalte von technologyscout.de finden Sie jetzt auch hier auf der Seite.

English

Hi, you might wonder why you got here instead of seeing the german version you expected. I realized, that it is a bit of a hassle to support two blogs with very similar content, so I merged them. So now you can find the content from technologyscout.de in here.