Voice on phone: Sir, On a scale of 8-10, how would you rate our service?
Me: I would not rate you on a scake of 8-10
Voice: Sir Why wouldnt you rate us on a scale of 8-10
Me: Because 8-10 is not a scale.
Voice: Ok, How would you rate us on a scale of 1-10
Me: Sorry I cannot rate since you have confused me completely now.
This is an actual conversation that happened with a car service station a couple of years back.
I am reminded of this incident because of a recent similar incident with an automobile provider.
There are some issues with my new car and I am getting continious calls from some call center asking me about my "purchase, marketing and delivery experience". When i tell them about my overall unsatisfactory experience which is still unresolved, they get confused and ask me to rate just the marketing and purchase experience. To cut the conversation short, I tell them I am overall unsatisfied but if they want feedback I can give it to them, but ask them to confirm if they really want it since my experience is negative. This normally either scares or confuses them and they say it is okay and hangup. Unfortunately some checklist somewhere shows that my feedback is not complete so I get periodic phone calls with the same questions and the cycle will continue till some one finally crosses my name from the checklist.
Ok What has this got to do with metrics and measurements?
This is a malaise that is spreading far and wide and shows how wrong measurements can generate wrong behaviours. In the above example, the person did not know if she should collect negative feedback and took the easy way out by just not collecting the feedback. In the process the company lost an opportunity to get genuine feedback and worse got a lot of contaminated feedback since it was incomplete. In the first case of the scale of 8-10 it was either fudging at its worst, or a complete defective program intended for self delusion or to fool the world.
Now the fault does not lie with the people who do the surveys but rather the management which is not clear about what and why they are measuring. Take for example a resteraunt. In case you fill in the feedback and mention that something was average, you have delayed your leaving for some time while the staff keep asking you why your feedback was "average" and not "good" and hinting at the fact that they will get into trouble with such feedback. In such cases I take the easy way out and just mark excellent for everything which again will waste a lot of other peoples time (and saving mine) while they collect, collate and analyze all the incorrect data and come to wrong conclusions.
The above were very simple and a "Dummies example" but this same problem is very prevalent in the field of software which is my bread and butter. Software and more so in the Indian context has some
charecteristics which make measurement very fuzzy. The whole measurement hoopla started when companies started chasing ISO and CMM and came to the conclusion that it was mandatory to collect metrics without understanding the intent. In the chase to get the coveted ISO certification or CMM assessment (many people confuse certifiation and assessment) , the metrics became the goal. In one of the software process discussion forums I used to frequent, a common question was "We want to start ISO/CMM/Sig sigma...) , so whar metrics should we collect"? The frivlous answer which I made on occasions was to ask "What are you trying to achieve?". If you are not sure, collect some easy metrics like the temperature, humidity..... If you know what you want to achieve, select measurements that will tell you if you are on track to reching your goals or not. However without knowing what you want to achieve, there is no point in measuring anything.
Collecting metrics for the sake of collecting metrics is a waste of time.
I have thought a lot on why we have still no clarity in the are of software measurements. There are volumes of books and papers on this subject, but for some reason I have very rarely seen a successful metrics program implemented which really used as a leading indicator for a strategic goal. I dont claim to be an expert or authority but do have considerable exposure to the industry and stand by my belief. I am not sure as to what the reason is for this gap.
Here are the some of the possible reasons:
1) Software is an art and not a science. Imagine if Leonardo Da Vince was managed by metrics. He would be tracking how many square feet of canvas he painted per day?
2) We are a bit fuzzy about what we do (ok this is an exagerration)
3) The whole measurement programs are designed by non practioners without any strong management sponorship, largely to satisfy some statutory requirement.I think this is the second most most common cause
4) The metrics are used to judge peoples performance. This in my opinion is the most common reason for a metric program failure. This is a very controversial subject with the majority of the view that how do you make people accountable for something unless they are measured. Now I dont have a very convincing answer for this exceppt to fall back on authorities and the fact that far more knowledablepeople and gurus say that metrics should not be used to judge people but are a tool to guide you to achieve objectives and
meeting the objectives is what should be used to judge performance. This is actually a whole discussion by itself and I will leave it at this. What is important to note is that practice of linking improvement metrics to individual performance is what generates wrong behaviours. A dummies example would be call centers where the employees I presume are measured by the number of calls they close per unit time. What is not measured is the percent of saisfactory resolution which I understand encourages the front end employees to close calls by just transferring to the next link which is normally more expensive. I cant think of crystal clear examples in the soatware arena and will leave it to another post.
I am looking for a forum to discuss this further but have got a bit rusty in this field. Anyway I hope this post generates some discussion since I am myself curious about this discussion.
What is important to note that measurements, metrics and KPIs are an absolutely invaluable tool for any sort of improvement or tracking and it is important to get this right.
