Tuesday, December 16, 2008

The Numerati and Pitfalls

American authors, true to their market-centred environment, are very good at coining terms and phrases to the constant despair and annoyance of the lexicographers at Oxford, I'm sure. Stephen Baker, author of The Numerati surely falls squarely into this category.

In Baker's book the Numerati are not some sinister, underground amorphous global conquistadors. Rather, they are geeky members of a global elite who are busy analyzing our every move. They're rummaging through mountains of data, looking for patterns of our behavior so that they can predict what we might want to buy, who we're likely to vote for, what job we'd do better than our colleagues. Some are even matching us with potential lovers. The Numerati are masters of symbolic realm. They're great at math and computer science. The Googleplex is crawling with Numerati. So is IBM.

Psychohistoric parallels
As part of the science fiction trilogy, the Foundation series (which was later expanded into a 5-part or 6-part series that included a prequel), Isaac Asimov's original protagonist, Hari Seldon conceived of a new psycho-science called psychohistory which was supposed to combine history, sociology and mathematical statistics to make (nearly) exact predictions of the collective actions of very large groups of people. Sounds rather like the Numerati, doesn't it?

.

It is the second book of the series, the Second Foundation, that I want to highlight. Hari Seldon had mapped out the different challenges that the new civilization would have to deal with in each phase of their development. Seldon had made these forecasts and projections via his skills with psychohistory. Every few decades, at a designated time, the leaders of the Foundation would gather before a vault that would open to reveal a holographic image of the oracular Seldon which would, then, guide the civilization through another few more decades of challenges.

But, in due course, a nemesis appeared that was not predicted by Seldon. This left the leaders of the Foundation in a dilemma that they had to deal with all by themselves.

Forecasted events and reality had diverged. There was a disconnect.

The asinine nature of assumptions
In any exercise involving the future, whether it is a basic SWOT analysis or a complex econometric model on where the flashpoint for the next world war will begin, assumptions have to be made.

This is where all crossroads meet, if you ask me. A mathematical genius with a poor understanding of historical nuances will make a different set of assumptions from a philosopher-historian. A religious zealot will make a different set of assumptions from a political idealist.

There is also a general tendency to place too much reliance on historical data even in situations of crisis and turmoil.

There is a failure to consider that a crisis is clear and incontrovertible evidence that all historical data should be suspect.

Would the Numerati have the wherewithal and wisdom to discard historical data in the event of a crisis? And, without such data how does one do forecasting?

Forecasting or fraud?
With this understanding it is no wonder that Malaysia's economic managers and, all economic managers the world over, keep trotting out last month's or last quarter's data.

But, the cardinal sin is to rely on the past data to paint rosy pictures of the immediate future.

No comments: