This week, I write in Pragati, the Indian National Interest Review about thinking beyond open data and creating a culture of openness in India:
Building sound public policies requires robust information systems and data. The state has the primary onus on provisioning for the public good that is knowledge – and traditionally the working of the Indian state involved secrecy as a core principle. The state enjoyed a monopoly on generation and access to large parts of knowledge and data relevant to public affairs, with its citizens often kept in the dark. Thanks to almost two decades of campaigns in the country, this paradigm has been overturned significantly.
When it comes to knowledge, information and data – the form of opening up determines how usable it is. Since the passing of the Right to Information Act into law, the focus of many advocacy efforts has shifted to usability and ease of access to government information and data. Release of information suo moto decreases the time cost of accessing information, and when it comes to data it is quite obvious that whether it is shared as a photocopy, PDF or an excel file makes a world of difference. Unfettered open access to data and information generated through taxpayer funds may remain a distant goal, but there are constant signs of progress. The Census of India is an exemplar of openness and friendliness as a data provider, and the Government Data Portal: data.gov.in was launched just last year is being populated with more datasets every month.
Increasing openness of official information is but one leg of improving the ‘public good’ quality of knowledge. Along with openness, the contestability of information has also increased in India over the last two decades. As Pratab Bhanu Mehta mentions, when official pollution numbers are not trusted, you have private organisations like the Centre for Science and Environment to monitor the pollution on their own and provide independent evidence.
Research institutions, not-for-profit organisations and the media are playing a larger role in generating data that is relevant to public affairs. Many private players routinely engage in data collection exercises, and conduct large surveys for research and to answer specific questions. Not only is this being done on a local scale, but nationally as well. Bangalore-based Public Affairs Centre conducted a national analysis of public services in India about a decade ago, and the ASER Centre provides state and district level information on children’s learning levels and education across the country, year after year. These are but two examples out of many.
While citizens of India are demanding more openness from their government and private entities are playing a greater role in contributing to public knowledge, openness as a culture has been far slower off the mark. Barring exceptions, research and not-for-profit organisations are far from open with their data in India today.
Data is collected with great care, cost and effort and is often used with great effect – but rarely more than once or twice. The original researchers often do not have the inclination, incentives or the luxury of going beyond their original mandates in analysis. Just like government data is underutilised if it only remains in an official report, the utility of privately generated data may far exceed this limited use.
Data and information are also network goods. Data sets can complement each other and together they can yield richer knowledge than they would on their own. A lack of public sharing of data sets and a culture that does not expect that of its knowledge creators prevents this from happening.
None of this is to say that individuals are not generous with what data is at their disposal. While rent-seeking remains a problem, many researchers and organisations are happy to oblige requests for data. However, this remains sub-optimal as personally investigating and enquiring after data results in very high costs for searching – if not monetary, then certainly in time and effort spent. The only way to change this is if more people adopt online, open disclosure of data.
While an open culture around data is desirable, it takes several complementary actions to get there. Access to private data, whether it is generated by a business or by an NGO, cannot be treated as a rights issue. Individual freedom and private property need to be respected while creating enabling incentives and encouraging voluntary efforts to open up information.
As research and data get exposed, the first problem that arises is that faults and errors become evident and can cause people to beat a quick retreat. What needs to be kept in mind is that pioneers who open their data sets before others can reap a large signaling dividend. Modest research with openness could have a higher impact than a superior quality study that remains closed. If sustained, open data researchers can build a reputation that is several notches above those who keep their data closed.
The second problem is that open disclosure results in a loss of control – where unknown anonymous users could use it without giving proper credit and for radically different ends. While this can indeed happen, the public benefit from open disclosure and the credit for it remains higher than such losses.
The third problem that can arise is that as the culture of sharing is weak and data science is still nascent in India, the benefits of opening up data sets may take a lot of time to yield impressive results. What is also needed is a kick starter – perhaps in the form of scholarships for students and researchers to use high quality datasets and publish new results.
Donors and foundations funding research and analysis also have an important role to play here – by committing to openness and providing platforms that enable sharing, they can change the work cultures in organisations that they fund. Official mandates to that effect government-funded research would work the same way. In India there is ample precedent for comfort: agencies such as the World Bank and others have embraced open data, it is for others to keep up.
If India is to transition to a true knowledge economy, open access, availability and contestability of public knowledge is paramount. A narrow campaign to liberate government data will have far less of a lasting impact than a broader attempt at creating a culture of openness and sharing around information and data. It is time we started on the latter.