Special report

The open society

Governments are letting in the light

|

FROM antiquity to modern times, the nation has always been a product of information management. The ability to impose taxes, promulgate laws, count citizens and raise an army lies at the heart of statehood. Yet something new is afoot. These days democratic openness means more than that citizens can vote at regular intervals in free and fair elections. They also expect to have access to government data.

The state has long been the biggest generator, collector and user of data. It keeps records on every birth, marriage and death, compiles figures on all aspects of the economy and keeps statistics on licences, laws and the weather. Yet until recently all these data have been locked tight. Even when publicly accessible they were hard to find, and aggregating lots of printed information is notoriously difficult.

But now citizens and non-governmental organisations the world over are pressing to get access to public data at the national, state and municipal level—and sometimes government officials enthusiastically support them. “Government information is a form of infrastructure, no less important to our modern life than our roads, electrical grid or water systems,” says Carl Malamud, the boss of a group called Public.Resource.Org that puts government data online. He was responsible for making the databases of America's Securities and Exchange Commission available on the web in the early 1990s.

America is in the lead on data access. On his first full day in office Barack Obama issued a presidential memorandum ordering the heads of federal agencies to make available as much information as possible, urging them to act “with a clear presumption: in the face of doubt, openness prevails”. This was all the more remarkable since the Bush administration had explicitly instructed agencies to do the opposite.

Mr Obama's directive caused a flurry of activity. It is now possible to obtain figures on job-related deaths that name employers, and to get annual data on migration free. Some information that was previously available but hard to get at, such as the Federal Register, a record of government notices, now comes in a computer-readable format. It is all on a public website, data.gov. And more information is being released all the time. Within 48 hours of data on flight delays being made public, a website had sprung up to disseminate them.

Providing access to data “creates a culture of accountability”, says Vivek Kundra, the federal government's CIO. One of the first things he did after taking office was to create an online “dashboard” detailing the government's own $70 billion technology spending. Now that the information is freely available, Congress and the public can ask questions or offer suggestions. The model will be applied to other areas, perhaps including health-care data, says Mr Kundra—provided that looming privacy issues can be resolved.

All this has made a big difference. “There is a cultural change in what people expect from government, fuelled by the experience of shopping on the internet and having real-time access to financial information,” says John Wonderlich of the Sunlight Foundation, which promotes open government. The economic crisis has speeded up that change, particularly in state and city governments.

“The city is facing its eighth budget shortfall. We're looking at a 50% reduction in operating funds,” says Chris Vein, San Francisco's CIO. “We must figure out how we change our operations.” He insists that providing more information can make government more efficient. California's generous “sunshine laws” provide the necessary legal backing. Among the first users of the newly available data was a site called “San Francisco Crimespotting” by Stamen Design that layers historical crime figures on top of map information. It allows users to play around with the data and spot hidden trends. People now often come to public meetings armed with crime maps to demand police patrols in their particular area.

Anyone can play

Other cities, including New York, Chicago and Washington, DC, are racing ahead as well. Now that citizens' groups and companies have the raw data, they can use them to improve city services in ways that cash-strapped local governments cannot. For instance, cleanscores.com puts restaurants' health-inspection scores online; other sites list children's activities or help people find parking spaces. In the past government would have been pressed to provide these services; now it simply supplies the data. Mr Vein concedes, however, that “we don't know what is useful or not. This is a grand experiment.”

Other parts of the world are also beginning to move to greater openness. A European Commission directive in 2005 called for making public-sector information more accessible (but it has no bite). Europe's digital activists use the web to track politicians and to try to improve public services. In Britain FixMyStreet.com gives citizens the opportunity to flag up local problems. That allows local authorities to find out about people's concerns; and once the problem has been publicly aired it becomes more difficult to ignore.

One obstacle is that most countries lack America's open-government ethos, nurtured over decades by laws on ethics in government, transparency rules and the Freedom of Information act, which acquired teeth after the Nixon years.

An obstacle of a different sort is Crown copyright, which means that most government data in Britain and the Commonwealth countries are the state's property, constraining their use. In Britain postcodes and Ordnance Survey map data at present cannot be freely used for commercial purposes—a source of loud complaints from businesses and activists. But from later this year access to some parts of both data sets will be free, thanks to an initiative to bring more government services online.

But even in America access to some government information is restricted by financial barriers. Remarkably, this applies to court documents, which in a democracy should surely be free. Legal records are public and available online from the Administrative Office of the US Courts (AOUSC), but at a costly eight cents per page. Even the federal government has to pay: between 2000 and 2008 it spent $30m to get access to its own records. Yet the AOUSC is currently paying $156m over ten years to two companies, WestLaw and LexisNexis, to publish the material online (albeit organised and searchable with the firms' technologies). Those companies, for their part, earn an estimated $2 billion annually from selling American court rulings and extra content such as case reference guides. “The law is locked up behind a cash register,” says Mr Malamud.

The two firms say they welcome competition, pointing to their strong search technology and the additional services they provide, such as case summaries and useful precedents. It seems unlikely that they will keep their grip for long. One administration official privately calls freeing the information a “no-brainer”. Even Google has begun to provide some legal documents online.

Change agent

The point of open information is not merely to expose the world but to change it. In recent years moves towards more transparency in government have become one of the most vibrant and promising areas of public policy. Sometimes information disclosure can achieve policy aims more effectively and at far lower cost than traditional regulation.

In an important shift, new transparency requirements are now being used by government—and by the public—to hold the private sector to account. For example, it had proved extremely difficult to persuade American businesses to cut down on the use of harmful chemicals and their release into the environment. An add-on to a 1986 law required firms simply to disclose what they release, including “by computer telecommunications”. Even to supporters it seemed like a fudge, but it turned out to be a resounding success. By 2000 American businesses had reduced their emissions of the chemicals covered under the law by 40%, and over time the rules were actually tightened. Public scrutiny achieved what legislation could not.

There have been many other such successes in areas as diverse as restaurant sanitation, car safety, nutrition, home loans for minorities and educational performance, note Archon Fung, Mary Graham and David Weil of the Transparency Policy Project at Harvard's Kennedy School of Government in their book “Full Disclosure”. But transparency alone is not enough. There has to be a community to champion the information. Providers need an incentive to supply the data as well as penalties for withholding them. And web developers have to find ways of ensuring that the public data being released are used effectively.

Mr Fung thinks that as governments release more and more information about the things they do, the data will be used to show the public sector's shortcomings rather than to highlight its achievements. Another concern is that the accuracy and quality of the data will be found wanting (which is a problem for business as well as for the public sector). There is also a debate over whether governments should merely supply the raw data or get involved in processing and displaying them too. The concern is that they might manipulate them—but then so might anyone else.

Public access to government figures is certain to release economic value and encourage entrepreneurship. That has already happened with weather data and with America's GPS satellite-navigation system that was opened for full commercial use a decade ago. And many firms make a good living out of searching for or repackaging patent filings.

Moreover, providing information opens up new forms of collaboration between the public and the private sectors. Beth Noveck, one of the Obama administration's recruits, who is a law professor and author of a book entitled “Wiki Government”, has spearheaded an initiative called peer-to-patent that has opened up some of America's patent filings for public inspection.

John Stuart Mill in 1861 called for “the widest participation in the details of judicial and administrative business…above all by the utmost possible publicity.” These days, that includes the greatest possible disclosure of data by electronic means.

This article appeared in the Special report section of the print edition under the headline "The open society"

The data deluge

From the February 27th 2010 edition

Discover stories from this section and more in the list of contents

Explore the edition