Tag Archives: open data

Open data comes of age

If you live in Italy and are curious about your local authority’s pattern of spending and taxing, you are in luck. Since last week, OpenBilanci publishes on the web detailed financial data from all the 8,092 Italian local authorities for the past ten years. Both budgets and closed ex post accounts are available, along with a galore of indicators like financial autonomy or spending velocity. Not only are all data downloadable and open: OpenBilanci sports a nifty web interface for preliminary data exploration. The latter is a feature found also on other highly successful Italian open data projects like the mighty OpenCoesione, that released spending data on 749,112 projects funded by the country’s cohesion policy. And no surprise: though OpenCoesione is a government initiative and OpenBilanci is not-for-profit one, the same team of visionary coders stand behind both projects, through both a non profit and a for profit arm.

In the space of only a few years, open data have become a formidable force for openness, transparency and even data literacy in a country that badly needs all three. Forward-thinking civil servants and political leaders in some of Italy’s 20 regions (and some cities) have been working together with civic hackers for years now: Lazio has funded OpenBilanci through its SME-centred innovation policy, whereas Emilia Romagna has successfully built a partnership with the largest Italian open data community, Spaghetti Open Data. In a veritable stroke of genius, the city of Matera has decided to host on its own open data portal any open dataset produced by the local community.

When public authorities do not play ball, Italian civic hackers simply proceed to open up government data anyway. One of my favourite projects in this sense is Confiscati bene, started during an epic Spaghetti Open Data hackathon. The group wrote a crawler to extract data from the (non-open) website of ANBSC, a government agency tasked with reallocating assets confiscated to Mafia bosses and other assorted mobsters (the Italian police is doing a sterling job there, since ANBSC is juggling over 11,000 such assets). It cleaned them up, geocoded them, made them downloadable, built the customary sleek interface for web exploration, embedded them into a brand new website and released everything as a gift to ANBSC. OpenBilanci itself entailed scraping over two million web pages.

I know Italy’s scene best, but exciting open data projects are appearing everywhere. My absolute favourite one is British: OpenCorporates gathers data on over 60 million corporations all over the planet. Using unique identifiers and information on ownership structure, OpenCorporates shines a light on the corporate world, that has far less tight legal requirements on transparency than government. This OpenCorporates-based visualization, for example, will teach you much about Goldman Sachs.

It looks like the open data movement has come of age. It was surprisingly fast: in less than four years we went from a small cadre of nerds obsessing on Tim Berners-Lee famous “raw data now” speech to a strong community (there are almost 1,000 subscribers to the Spaghetti Open Data mailing list, churning out twenty messages a day 365 days a year) and a phalanx of young decision makers that understand the issue and are plugged into the community. I am proud of you all, my sisters and brothers in arms. And the best is yet to come – especially as we come together all across Europe, as I am sure we will soon since the times are ripe for this to happen. Who knows, data culture might even be able to shift European politics away from populism and onto evidence-based debate.

Photo: Marco Giacomassi

Missing out: why we don’t have an European open data community (yet)

The last weekend of March was SOD14, the second yearly gathering of the Spaghetti Open Data mailing list. The acronym in English may be awkward (it was just too funny to pass on!), but the event was just great. We had 182 people registered over the three days; attendance peaked at the conference on Friday 28, with 139 people in the room at the same time. About 100 people attended the hackathon Saturday 29 and the training session on Sunday 30. We produced 12000 tweets (and, being geeks, we archived them all). Everyone came on their own time and money.

The hackathon was spectacular: we had planned for four tracks, but so many people showed up that we ended up doing seven. We hacked things like data on goods confiscated to mafia bosses, the Open Knowledge Foundation’s open data census; we designed a sort of peer-to-peer service for civil servants wishing to release open data; there was a track for lawyers and one for civic monitoring.

Everything , from conference program to hackathon tracks, was built from the bottom-up. Spaghetti Open Data is a community: it has no money, no corporate structure, no leaders, so it can’t help being bottom up.  SOD14 was completely organized by volunteers: though our host city of Bologna and its regional government stepped in with free venues, free coffee and flawless connectivity and two (community-designed and delivered) mini-courses, for a grand total of 1500 euro. The community provided video trailers, logos, jingles and ringtones, t-shirts,stickers and even superheroes; there was a very diverse attendance, (data geeks, data lawyers, developers, data journalists, policy makers, even some open data archeologists) with a strong female presence. SOD14 had the playful energy of the really grassroots events. And when the event was over, people simply retreated to the mighty mailing list: at the time of writing, Spaghetti Open Data has three and a half years of life, 894 subscribers, 1,840 threads, an estimated 20,000 posts (well over 20 a day in 2014). It is far and away the largest open data resource in the Italian language.

So all was well, except that something was missing. There was no Europe in SOD14.

We did our best to stay in touch with our European brothers- and sisters-in-arms. We had our only keynote in English – with Wikimedia Germany’s Adam Shorland telling us about Wikidata. I personally called EPSI, DG CNECT’s initiative for promoting open data across the European Union, and asked them for support – not in the form of money, which we can’t accept anyway, but embodied in someone to come to our gathering and say “you are not alone, we are happy you are doing this work”. Even though we had updated and verified the EPSI scoreboard for Italy during 2013, nobody showed up at SOD14 to say “thank you” in person: they agreed to do so initially, but then they decided they were covered by Matteo Brunati, EPSI’s correspondent for Italy, present at SOD14.

Dear European Commission, as a European patriot and  an open data activist, I feel it is my duty to let you know you’ve wasted an opportunity, and to advise you never to do that again. In SOD14 we were not discussing Italian open data problems. All our problems were at least European. For example, we had a fascinating session about open data in archeology and cultural heritage. Italy is hardly the only European country to deal with these kinds of issues; we are struggling with very conservative cultural institutions here, and could benefit a lot from comparing notes with people doing equivalent work in, say, Greece or France. That’s where you could have made a difference – but didn’t. I could make ten more examples like this from SOD14 alone, and so could you.

Matteo is a high-level civic hacker, and EPSI is very fortunate to have him on board. We, on the other hand, are his home community, and talk to him every day. There is no value added to our event if you just put a different hat on his head. The way you add value to Matteo’s European commitment is to dispatch him to events like ours in Estonia, Belgium or Ireland; and the way you add value to Italian events like SOD14 is to dispatch people like Matteo, but with experience in Denmark and Spain and Austria. It’s horizontal relationships that make a community. I know you know this, because you have been doing Erasmus-like stuff in many variants and for a long time. But horizontal relationships are slow to build, and no one is working on building them now – not even you. And so, things that should be taken for granted don’t happen. Why don’t we have civic hackers from across the continent cooperating in doing some open data project about the European elections? Because European civic hackers don’t get the chance to hang out together all that much. Even TweetYourMEP was built exclusively by Italians. So, there is no such thing as a solid European civic hacking community.

But don’t give up just yet. Europe played a key role in unlocking the supply side of the open data scene. The EPSI Directive was fundamental in nudging less data-friendly governments like ours onto the right path. Europeana is a great idea. You have done well on those fronts: why should you not do equally well in helping unlock the demand side of open data? A year ago, EPSI interviewed me and asked me “what do you think Europe should do around open data?”. And I replied “invest in the community. Give them free venues, free travel and something to do” (this video, at 6:08). I still think that would be the best way to use your EPSI infrastructure. Actually, tell you what: why don’t you go all the way and start an “Erasmus for Open Data” program. A few hundred international exchanges, with people from across the continent actually working together on data projects, would go a long way towards creating the small world network we need to be a community at the European level. Spaghetti Open Data stands ready to help. Are you game?

“Not so bad, actually”: an interview of the state of open data in Europe

Update: on July 30th EPSI Platform has taken down the video to perform some editing of the captions. An updated version should become available presently.

A few weeks ago, as I went through Barcelona to participate in the Open Government Day, I had a chat with Montse Delgado on open data in Europe and some of my projects, notably Edgeryders and OpenPompei, which both originate in the open government paradigm. Montse and her colleagues got the chat on video and released it on the EPSI platform website, the closest thing we have to an official European open data discussion space. The whole thing can be found here.