Tag Archives: World Bank

Open data: what if we used Wolfram Alpha?

In several countries governments and local authorities are starting to release their data for citizen reuse and remix. This is important in itself, but clearly these data will generat more impact the easier and intuitive using them is. I am seeing very different usability strategies being tried out.

At one end of the spectrum, some authorities go for “in-house” visualization. An example is the OECD with their eXplorer: it comes with a sophisticated visualization interface that supports animations, multilayer maps, Google Maps integration. The only problem is that the data themselves are locked inside, as are citizen-created visualization (“stories” in eXplorer parlance). Just about the pnly thing you can do is export them as XML files to share with your friends, but they need to go to the eXplorer website to see it. In general, the system is complicated and lacks flexibility; plus, it’s quite unfriendly for beginners (the instructions files is more than 30 pages long).

At the opposite end we find experience like the Italian State Accounting Service’s. The databases are downloadable, and there are instructions to generate tables summarizing them. Unfortunately, they are very specific, assuming as they do that all citizen use a specific function (pivot tables) of a specific software, proprietary and expensive on top of that (Microsoft Excel. It would probably have been more tasteful to refer the tutorial to Open Office). Except for expert users of Excel, this system is “all or nothing”: either you are looking at enormous, unmanageable disaggregated tables or you invest several hours to follow the tutorial and try a few ways to crunch the numbers so that they make sense. It’s ok for researchers, but it does not create an interest for citizens to play around with data.

Maybe a good middle ground is the World Bank’s strategy. World Databank enables the creation of simple reports (charts and maps included) directly on the site, and it also allows users to download the data in several formats. So a citizen can start by exploring directly on the website in less than a minute she can be looking at a simple chart. Then, if she so chooses, she can download the data and explore them more thoroughly with the software of her choice.

I think open data policies are just as meaningful as the community of citizens that know how to interpret, reuse and explain to others public data is wide. On this ground, I would strongly advocate to public authorities contemplating them to incorporate in their websites rapid preview features, as the World Bank did.

Developing those feature can be expensive (eXplorer certainly does not look cheap). A low cost possibility is probably to write a widget for Wolfram Alph, the computational engine of the man who gave us Mathematica. Its computational capacity is more than adequate to any reasonable open data use (it does some amazing stuff: try typing “compare a mouse and an elephant” as in the compute field): the problem is rather interfacing it with government databases. If that can be done, it becomes easy to write intuitive widgets like the one above – with the added benefit that the data then become accessible to anyone using Wolfram Alpha anywhere in the world, even if they are not even aware of the existence of this database in particular. More open data than that…

Wolfram Alpha could provide public authorities with a fast, cheap preview mode for citizens to develop a taste for open data; this would also increase the value of Wolfram Alpha itself, as the mass of data accessible to the engine is increased. It’s not for free, though: Wolfram sells subscriptions that cover a certain number of monthly queries. I wrote them to find out whether they have solutions for this kind of client, curious to see what they get back to me with. The real solution would be an open source version of Wolfram Alpha, but I am not aware of anything remotely like that.

Open data: e se usassimo Wolfram Alpha?

In molti paesi (e finalmente anche in Italia) le pubbliche amministrazioni cominciano a rilasciare i loro dati perché i cittadini li riutilizzino e li remixino. Già l’atto è importante, ma naturalmente questi dati genereranno tanta più energia sociale quanto più semplice e intuitivo sarà il loro utilizzo. Le strategie di usabilità che ho visto in giro sono molto, molto diverse tra loro.

A un’estremità dello spettro, alcune organizzazioni puntano sulla visualizzazione, per così dire, in-house. Un esempio è l’OCSE, con il suo eXplorer: un’interfaccia di visualizzazione sofisticata, che permette animazioni, mappe multistrato, integrazione con GoogleMaps. L’unico problema è che i dati rimangono bloccati lì dentro, e anche le visualizzazioni – il sistema le chiama “stories” – si possono vedere solo da eXplorer. L’unica cosa che puoi fare è esportarle sotto forma di un file XML da condividere con i tuoi amici e colleghi; ma poi loro devono caricarlo su eXplorer per poterlo leggere. In generale, il sistema è complicato e poco flessibile; inoltre, lo scalino per gli utenti novellini è piuttosto alto (il file di istruzioni è oltre 30 pagine).

All’estremità opposta ci sono esperienze come quella della Ragioneria Generale dello Stato italiano. Il database è scaricabile, e ci sono istruzioni per generare tabelle riepilogative. Purtroppo, sono molto specifiche: presuppongono che il cittadino usi una funzione specifica (tabelle pivot) di un particolare software, per di più proprietario e costoso (Microsoft Excel. Forse sarebbe stato più elegante riferire il tutorial a OpenOffice). Tranne che per gli utenti esperti di Excel, questo sistema è “tutto o niente”: o ti confronti con enormi tabelle di dati disaggregati o investi una mezza giornata per seguire il tutorial e provare a fare qualche ipotesi di aggregazione. Va bene per i ricercatori, ma non crea interesse per giocare con i dati in chi ricercatore non è.

Forse una buona via di mezzo è la strategia della Banca Mondiale. World Databank permette la creazione di semplici report (compresi grafici e mappe) direttamente sul sito ma consente anche di scaricarsi i dati in formati diversi. Così, un cittadino può fare una prima esplorazione direttamente dal sito: in un minuto scarso può già guardare un semplice grafico. Se poi ci prende gusto, scarica i database e costruisce le elaborazioni che preferisce con il software che preferisce.

Credo che il senso delle politiche di open data sia tanto più profondo quanto più si riesce ad allargare la comunità dei cittadini che sanno capire, riutilizzare e spiegare agli altri i database governativi. Per questa ragione consiglio assolutamente a quelle autorità pubbliche che volessero intraprendere questa strada di incorporare nei loro siti delle funzionalità di preview rapida, come quelle di World Databank.

Sviluppare queste funzionalità costa caro (eXplorer certo ha l’aria di costare caro). Una possibilità low cost è probabilmente quella di scrivere un widget per Wolfram Alph, il motore computazionale ideato dall’uomo che ci ha dato Mathematica. Le sue capacità di calcolo sono largamente adeguate a qualunque ragionevole uso in questo contesto (fa delle cose semplicemente sbalorditive: provate a inserire nella riga di ricerca “compare a mouse and an elephant”): il problema è piuttosto quello di aggiungere a fare comunicare Wolfram Alpha con i database governativi. Se si riesce, però, è possibile scrivere widget facilissimi da usare come quello qui sopra, senza contare che a quel punto i dati diventano accessibili a chiunque, in tutto il mondo, usi Wolfram Alpha, anche se non lo usa dal sito dell’autorità in questione — e che i widget possono essere facilmente copincollati in altri siti e blog. Più open data di così…

Lavorare con Wolfram Alpha permetterebbe alle autorità pubbliche di avere una modalità di preview interattiva in tempi rapidissimi, e con investimenti molto piccoli in sviluppo software; ed accrescerebbe il valore di Wolfram Alpha stesso, che naturalmente è funzione della massa di dati a cui riesce a accedere. Non è gratis, però: si pagano abbonamenti che coprono un certo numero di interrogazioni mensili. Ho provato a scrivere all’azienda chiedendo se hanno una linea sugli open data, vediamo se mi rispondono e cosa. La soluzione vera sarebbe una versione open source di Wolfram Alpha, ma non ho notizia di cose lontanamente simili.

Black swans and banking regulations

Contrordine Compagni hosts a brilliant post contained in a World Bank blog called Crisistalk, created as a space for the Bank to contribute to the debate about the financial crisis, then in full rage. That blog has since been taken down: I think this post must be preserved, as it conveys the (rather stormy) climate in financial institutions at the time. So I (Alberto) volunteered with Ryan Hahn, blogger-in-chief at the Bank’s Private Sector Development blog, to host it myself. While unable to give me formal permission, Ryan has been quite encouraging: I mean well, after all. Should the author or the World Bank Group have any issues with it, I will take down the post immediately.

The post was published by John Nellis on 2 March 2009. Nellis was a Senior Manager in the World Bank’s Private Sector Development Department. He is now Principal of the consulting/research firm, International Analytics.

On February 24 I attended the first day of a two and half day World Bank conference on Markets and Crises: What Next and How? Two sessions were particularly interesting: the keynote address by Nassim Taleb, author of “The Black Swan: The Impact of the Highly Improbable,” and a panel on “What has the financial crisis taught us about risk management?” with Mark Carey from the Federal Reserve Bank of the US, Stijn Claessens from the IMF, Martha Cummings from Banco Santander and Mark Zandi of Moody’s.

First, Mr. Taleb. He is wildly entertaining and delightfully iconoclastic; he left no one in or outside the room uninsulted. All economists are worthless. Certain central bankers are charlatans and fools. French bankers are the worst of all possible bankers. No professor of finance has ever been right about anything (especially those who for years kept rejecting his submissions to finance journals). No one on the staff of the New York Times knows anything about finance or economics or statistics. Regressions are useless; models are worse. “People who use history as a guide do not understand history.”

The last is the essence of his central thesis — the common and fatal error of those working in and around economics is that they generally assume that what happened yesterday is highly likely to occur again today; they see and model the world in terms of “Gaussian” statistics; that is, the normal distribution. But his study (of forty years of data on things that have been priced) leads him to see that in finance and economics “most of the variance is concentrated in a very small number of events.” (This is measured by “kurtosis,” a term in statistics meaning the extent to which rare extreme deviations from the norm explain more of the variance than frequent smaller movements around the norm. Phew!)

So, convinced that “as risks mount fewer and fewer outsized events are required to bring the whole system down,” he some years back predicted doom and started shorting the market, day after day after day, losing money day after day…..etc. Until last year. Bam. With the market gains (and the loot from two million copies of the book sold so far) he is now invited to World Bank seminars to point out with gusto their (our) ignorance and culpability, and everyone else’s too. I got the distinct impression that he is happy with the money but much more enthralled by the vindication against all those who called him a kook, a cultist, or worse. He is having a grand time. And he was right about the crash.

When it comes to the question “So what should we do now?” he is not helpful. Questions included, “How should we go about reforming the regulatory system?” Answer: Don’t try to reform it; scrap it. “But what should we do?” Answer: Abolish debt. All of the world’s great religions agree; there should be no interest. “Did regulators make the crisis worse by their incompetence?” Not an area of interest to him; he is not concerned with minor tinkering with the system. Good-bye.

As I said, rewarding and fun, but of little value to those trying to work their way out of the present mess.

The afternoon seminar was toute autre chose: four earnest, smart, articulate, wonkish insiders admitting some errors and looking hard for practical solutions to an increasingly frightening debacle. But at the end of the session the practical answers were notable by their absence.

Carey, from the US Federal Reserve Bank, argued that bank regulators such as himself had certainly seen signs of problems but thought they were off-white or grayish swans, not black. Small reform would, they hoped and thought, suffice. He stated that this did not derive from a dependence on misguided models, but rather that avoidance of the dramatic is ingrained in financial regulators; that those in central banks must be absolutely sure they are looking at a bubble before trying to use monetary policy to burst it. Knowing in advance that a problem area is a real bubble is not all that simple. And they must also be sure that calling attention to a possible bubble does not create the very situation they are trying most to prevent — a panic. He concluded that increased regulation by itself will not solve the problem; what has taken place is “not a technical problem, but rather a governance problem.” (The distinction was not made clear.)

Martha Cummings has excellent credentials; as head of Risk Analysis at Banco Santander she advised her board that much of the portfolio was incomprehensible and she recommended getting out of lots of loans whose underlying assets were based on impenetrable securities. Thus, Banco Santander is today not in such a poor position as many other banks. She was tougher than Carey; she says we should have seen it coming. Masses of people were and still are living far beyond their means, in debt up to their eyes and leveraged to the hilt. We did know it was coming; we just didn’t know when. We kept thinking we would somehow get 24 hours notice and get out ahead of everyone else—and we were wrong. We knew banks were doing deals to get market share, regardless of profitability; we depended on ratings agencies and models, not facts and basics. We consistently ignored the signals because “no one wanted to end the party.” We ignored a basic fact: “The market can stay irrational far longer than you can remain solvent.” Her advice is on the ruthless side: let housing prices fall to where the market wants them to be; let the lenders and the borrowers take the hit. Trying to prop it all up will fail and make things worse. Let it rip (at least she has a clear point of view).

Mark Zandi of Moody’s first flagellated himself and then all other supposed supervisors of the system. “Both the architecture and the plumbing of the regulatory system were deeply flawed.” No one in the oversight system had responsibility for saying “this is a bad loan.” The lender wanted a fee and passed the mortgage to someone else. The investment bank securitized the loans, took a percentage, and forgot about it. Rating agencies are paid by the lender, not the investor, and they “keep the ball rolling.” Bank regulators failed to recognize or call attention to the problem. And here we are.

All agree: “We forgot the basics. We took a crisis and made it a panic.” And all agree that while more and more dense regulation is now inevitable, they doubt it will really change the essentials. The Basel regulations had little or nothing to do with the crisis, either in terms of abetting or hindering it. Carey said that his many years of experience fitted him to sense, within minutes, a bad bank, but he had long ago been promoted out of bank examination. And if they sent him back to it he would quit; Bank examination is a lousy job, now handled, worldwide, by very young and inexperienced people. The panel agreed that if corporate boards had members who fully understood risk models they might demand sufficient information, examine it carefully, and instruct management on how to act. An unlikely prospect. They all are, to a varying extent, concerned that a new thicket of regulation will slow or halt a quick recovery, but Zandi (of Moody’s) went on record as saying that the Federal Reserve has the capability and responsibility of forecasting bubbles and being much more aggressive on informing the public of their prognostications.

All this was much more depressing than Mr. Black Swan.