An Open Data Discussion

By Peter McComb and Malene Felsing

What are open data?

The New Zealand government defines open data as data that anyone can use and share - data which have open licences, are openly accessible and are both human- and machine-readable1.

Following the example of numerous countries, in August 2017 the New Zealand government adopted the International Open Data Charter, a non-binding agreement mandating that government data are open and accessible to all. The Open Data Charter builds on the New Zealand Declaration on Open and Transparent Government and the Data and Information Management Principles, through which the government has communicated an expectation that agencies proactively release high-value data, and work towards an ‘open by default’ approach2.

These policies were adopted based on evidence from overseas that open data enables the development of new knowledge, tools and services, which drive economic development.

Benefits of open data

Since the inception of the term ‘open data’ in 1996, and the concurrent explosion of the internet and the movement towards open source code, the benefits of open data have been assessed by many. The results of the research show that data create more value when they are widely utilised and well-governed.

The clear benefits of open data include:

  • Increased innovation as data become accessible to users from different disciplines.
  • Reduced barriers to entry into markets.
  • Creation of economic value through the development of new products, services or activities.
  • Efficiency gains in the public sector as agencies gain access to data that help streamline operations, and through non-duplication of data collection efforts.
  • Improvement in efficiency and productivity of private businesses using the data.
  • Flow-on effects as emerging second-order users add further economic and social benefits to the economy.
  • Increased government tax revenue through expanded economic activity, as well as higher revenue for individual agencies through the sale of high-value information to companies.
  • Public engagement as the wider population can access educational and cultural knowledge.
  • Improved social welfare as society benefits from transparent and accessible information, stimulating collaboration, participation and social innovation.
  • Greater transparency and accountability of public service providers.
  • Better policy-making based on better data.

There are numerous examples of international macroeconomic studies into the economic impact of open government data. A 2014 report3 assessing the value of open data for G20 economies predicted an AU$19 billion return on investment over five years from doubling the accessibility and use of Australian government and research data.

Who benefits the most?

Most analyses agree that open data benefits everyone - there are no real losers and widespread wins, because open data makes better use of existing resources. Consumers and the broader society stands to win the most, although there are some benefits to data providers. A European Union study4 concluded that releasing public sector information leads to modest direct revenues to governments. It estimated that most European nations would see a gain of 1% of agency budget, but that gains of up to 25% were possible. These estimates were based on the Netherlands and the UK, who both gather revenue from opening up government data. However, all studies agree that that open data creates new types of businesses, providing opportunities for small and medium enterprises and business models like advertiser-pays rather than end-user-pays5.

Open weather data

Of all the different types of public sector data, weather data have been identified as particularly important because everyone - including individuals, private companies, local and national government departments - can all benefit from it. Applications that make use of open weather data can therefore potentially have a huge impact.

Several studies have attempted to place a value on open weather data. Research from 20096 indicated that US adults used more than 300 billion forecasts per year at the time, which was valued (by the general public) at $286 per household per year, providing an aggregate annual valuation of weather forecasts of about US$31.5 billion. At the time, government and private sectors spent US$5.1 billion on meteorological operations, research and forecasting, which means that the value provided by weather forecasts was 6.2 times higher than the cost of producing them.

Open weather data in New Zealand?

This is a really good question for New Zealand to ponder during 2018. We have a small economy with a big ocean to manage and we tend to get a lot of weather! There are two state-funded organisations independently collecting data and independently producing high quality national weather forecasts, plus a range of international weather forecast providers with various types and qualities of data available. So, what is our current source of weather truth? Is there a better way for the needs of the New Zealand economy and general public to be served? To answer that question in a meaningful way, it’s helpful to consider where exactly data comes into the equation.

 The value ladder for society starts with data. We need open data so that all sectors of the economy can contribute to the generation of information and knowledge, so ultimately we can collectively and consistently make wise decisions.

The value ladder for society starts with data. We need open data so that all sectors of the economy can contribute to the generation of information and knowledge, so ultimately we can collectively and consistently make wise decisions.

This figure shows that data is the start of an entire value chain that takes society to a place where we can (hopefully) make wise decisions. In weather forecasting, data is both the observations and the model outputs. However, ‘data’ by itself is nearly useless to people, but when we turn it into ‘information’ such as a map or a graph, we create value because the data gains context. ‘Knowledge’ comes about when that information can be put to use, such as a warning of potentially hazardous conditions developing. It is situational awareness and requires prior experience or empirical findings. Finally, ‘wisdom’ is the place we want to get to in society, and modern-day tools like social media, smartphones and web sites allow us ready access to knowledge in its myriad of forms. The wise decisions we expect individuals to make are ones like:

               “I’m not going fishing today because the bar crossing might become dangerous.”
               “The conditions for climbing the peak are perfect tomorrow, so let's wait one more day.”
               “Better move the stock out of the river paddock because it's likely to flood tonight.”

Traditionally, the creation of weather knowledge has required human intervention, with trained forecasters interacting with data to produce guidance. As numerical weather models become better and more detailed weather measurements become available, the ability for models to accurately quantify and predict the weather will continue to improve. Weather on planet earth is still a chaotic process, however combining computer weather models with machine learning and other such techniques will allow us to become more quantitative when describing the present and future weather states.

And quantitative data is exactly what modern infrastructure demands. Think automated smart irrigation systems, traffic management algorithms and transport logistics networks to name just a few. All require weather data in the raw form to ingest and create information and knowledge specific to those applications. The global market for this is enormous, which is why private sector worldwide has invested so heavily in weather prediction and a whole new generation of sensing technologies. Within a few years the capability of private weather services will likely exceed that of most national weather agencies. Couple that with the disruption from constellations of small low cost weather satellites, and we have the imminent arrival of a new world order in weather forecasting.

What next?

So, back to data. New Zealand doesn’t have a great track record when it comes to making public funded environmental data openly available - but it's true we are making steady progress (see here). At the same time, the quantitative demands of the modern world are resulting in a flood of high quality private data becoming available and strong consumer options regarding alternative weather sources. In order to remain relevant, it might be timely for New Zealand to clearly define the national weather ‘sources of truth’, and perhaps actively push data into the economy to gain the universally benefit from the wealth multipliers that arise when enterprise creates knowledge from data.

Note - these are views of the authors, and do not represent MetOcean Solutions or MetService.

____________________

  1. https://www.data.govt.nz/toolkit/what-is-open-data/
  2. https://www.data.govt.nz/assets/Uploads/Adoption-of-the-International-Open-Data-Charter.pdf
  3. Omidyar Network (2014) Open for Business: How Open Data Can Help Achieve the G20 Growth Target https://www.omidyar.com/sites/default/files/file_archive/insights/ON%20Report_061114_FNL.pdf
  4. Vickery, G. (2011) Review of recent studies on PSI re-use and related market developments. European Commision. Available at: https://ec.europa.eu/digital-single-market/news/review-recent-studies-psi-reuse-and-related-market-developments
  5. Deloitte (2011) Pricing of Public Sector Information Study (POPSIS) - Models of supply and charging for public sector information (ABC) - final report. Available for download at: https://ec.europa.eu/digital-single-market/en/news/pricing-public-sector-information-study-popsis-models-supply-and-charging-public-sector
  6. Lazo, J.K., Morss, R.E., and J.L. Demuth (2009) 300 Billion Served: Sources, Perceptions, Uses, and Values of Weather Forecasts. Bulletin of the American Meteorological Society. 90(6).