My brothers- and sisters-in-arm of Ondata have been exploring ways to use LLMs to facilitate their interactions with online sources of open and statistical data. In so doing, they have hit on a working style that I find very promising.

The obligatory disclaimer: I do not endorse “artificial intelligence”, a highly controversial socio-technological construct. As an economist, I am irked by the economic aspects of the tech: the insane concentration indices in this industry, the information asymmetries, the gigantic losses associated with the expansion of its customer base, its “dogshit unit economics”, to quote Ed Zitron, its massive environmental and social costs, its dark politics that makes it gravitate towards disinformation and deepfakes. I could go on, but you get the idea. This is probably preaching to the choir anyway, given that most people I hang out with are very skeptical of anything that smacks of “AI”.

And yet, Andrea Borruso and Matteo Fortini (among, undoubtedly, many others) have proposed an approach that I quite like. It goes like this: if you ask a data-related question to a LLM, such as “is the unemployment rate in Guatemala growing faster or slower than that in Ethiopia?”, you get better than even odds that the LLM will just make something up. A more reliable approach is to download the pertinent data from an authoritative source, and see for yourself. Andrea and Matteo, as honoured elders in the Italian open data community, are no strangers to tracking down data warehouses, poring over API documentation, building structured queries and feeding the result of those queries to scripts that will tease answers out of the data. And their substantial experience tells them, inequivocally, that having to do this work every time you want a question asked sucks big time.

But they found an alternative: codifying their deep knowledge of structured data and structured metadata in ways that are legible to an LLM. Then, instead of asking the LLM the direct question, they can instruct it to look up the data at the source – say, the World Bank’s data warehouse. This is possible because, after 20 years of open data movement, authoritative data to address many of those questions exist, and are structured. For example, there exists an ISO standard for machine-to-machine exchange on statistical data, called SDMX and followed by many statistics agencies. So, Andrea built OpenSDMX, a command-line interface that interacts with SDMX-compliant data sources. Give it a search term (unemployment rate Guatemala), and it will search the catalog, find the relevant datasets based on the metadata, and return a menu of options of what to download. For example, do you only want to know the most recent estimate of the unemployment rate, or would you like a time series? If sub-national data are available, do you want them too? Perhaps a disaggregation by sex, or age group? And so on.

OpenSDMX works well on its own, but it was set up so that you can interact with it via any LLM. This is done via something called a skill, which is a human-readable file containing the instructions necessary to use the command line interface well. To a human it reads like good documentation: to a LLM it is a long, detailed, well structured prompt. Thanks to it, LLMs can turn the instructions of their human operator into a structured flow of interactions with SDMX-compliant data sources by means of well-formed SDMX queries – there is no need to guess, because the skill contains correct, unambiguous structures written and validated by someone like Andrea and Matteo, who knows the standard well. Further specifications of the skill (more text files) address the quirks not disciplined by the SDMX standard itself. For example, while EUROSTAT (the European Union’s statistics agency) and ISTAT (Italy’s) both comply with SDMX, the former labels its variables in lowercase, and the latter in uppercase. This is the type of quirk known for causing lots of mild irritation to data scientists: having it encoded in the skill means your LLM benefits from the experience of veteran data scientists, even if you do not bother to read the skill.

The approach works not only for statistical data, but for any kind of data and metadata as long as they are exposed according to a standard. Another conceptually similar project built by Andrea is a Model Control Protocol (MCP) server to interact with open data portals and their content. This is possible, again, because the information therein is structured according to well-documented standards. Most open data portals are built on CKAN, and the datasets hosted therein follow another standard, DCAT. The logic is the same: open data portals are somewhat standard, and over the years the community has produced lore about how to whisper to them so that they will return the data we want. Encode that lore into text files that an LLM can interpret as a prompt. Then use the LLM as a natural-language interface to the data.

In social and economic research, it happens quite often that you have a hunch that can, in principle, be validated, or disproven, by looking at data. Andrea’s and Matteo’s approach greatly reduces the distance between formulating a question and getting an initial answer. It is even more useful when you have not a question, but a hunch, and you want to get an initial look at the raw data to see if you are even remotely right.

I decided to test OpenDMX as a way to validate or invalidate a conjecture. My conjecture was:

Within the European Union, Eastern European member states are undergoing more economic upheaval than average.

I wanted to test this conjecture by comparing the rates of business creation and business destruction (as captured by the creation of new companies and the cessation of existing ones). Furthermore, I want to do it by looking at sub-national-level data, if they exist. The idea is that upheaval is in the extremes, not in the averages: I am looking at the location of regions where business creation and destruction are unusually fast. To be clear, this is a completely unfounded conjecture, but it is a conjecture of the kind that economists make all the time, so it is valid as a test.

I launched an LLM and called up the skill, then wrote a prompt:

This gets me:

Good to know I can make a sectorial analysis by NACE codes, but no need to make this complicated right now. I want to see if there are macro differences between regions. For the same reason, I am not going to go into dynamic analysis, just get a snapshot. If I don’t see anything there is probably nothing to see, and then no point splitting hairs by doing sophisticated analysis on barely perceptible phenomena. So:

And I got:

After some beautification I got a scatterplot that tells the story I was looking for.

My hunch, it turns out, is not supported by the data. If anything, it is northern regions in the EU that are experiencing a high churn. Those green dots towards the right of the chart represent Estonian territories, with a Finnish one that is an outlier and not even shown. But this is not the point here. The point is this: investigating this hunch required me to think like an economist. I can formulate a testable hypothesis. I know what economic data can and cannot say. I have learned to stay away from overinterpreting them. The LLM here thinks like a data scientist, borrowing Andrea’s expertise. It knows how to look inside the EUROSTAT data warehouse, read its labels, and build well-formed queries in SDMX format. It also has access to Python libraries for simple data visualizations. This means that, in minutes, I can get to a very rough test of my hunch.

That makes me a centaur. I am borrowing the word from Cory Doctorow. He thinks that LLMs can be used to empower humans to do better and faster things that humans do, or, viceversa, to boss over humans so that LLMs themselves can do things that they themselves cannot do. In the first case, the composite of human and machine is a centaur, a human being running around with the strength and speed of a machine. In the second, it is a reverse centaur, where the human part of the composite is a meat appendage to the machine making all the decisions, or an accountability sink, there to take the blame for when the LLMs screw up. I am a centaur here, whereas an Amazon delivery driver is a reverse centaur, with the machine part micromanaging the human and forcing him to pee in a bottle so as to squeeze the last cent of value from his labour.

Since well before LLMs, the open data movement believes that, by taking care of data (collecting them with intent, documenting them, publishing them with open license, creating tools that people can use to access and process them), you could empower people to do more. An important category of people that we cared about empowering are journalists, researchers, activists: whoever needs knowledge to make a difference. These new LLM-compatible tools come from this lineage. An investigative journalist can now type a natural-language question, such as “which datasets do we have about flood damage in Latin America?”, and the CKAN MCP server will trigger a series of queries that will look for CKAN-powered open data portal, and then look within their catalogs for appropriate datasets. It will even do this independent of the language that the journalist is typing in, and that the datasets are documented in.