In its Intended Nationally Determined Contribution (INDC) to the UNFCCC as part of the Paris Agreement, India committed to 40% cumulative electric power capacity from non-fossil fuel resources by 2030. In June 2018 installed renewable generation capacity was 69 022 MW, equivalent to 20% of total capacity. If nuclear and hydro are included, the share of non-fossil fuel capacity increases to 35.3%. With a target of 175 GW of renewable capacity by 2022, this data would suggest that India is comfortably on target to meet its INDC target. And it will.

In 2017 India added more renewable capacity than conventional generation, yet simply adding new renewable capacity is only half the solution. Without investment in associated transmission and distribution infrastructure the benefit of the incremental renewable capacity is not realised and the rapid accumulation of new renewable capacity is exposing the limitations of India’s aged transmission infrastructure.

With 60 GW of wind and 100 GW of solar capacity planned for 2022, India will become increasingly reliant on intermittent generation capacity. This will require transmission companies to find a balance between renewable and thermal power to ensure uninterrupted supply. Additionally, it will require improvements in demand and procurement forecasting, both from a financial point of view and from a technical standpoint, as the increasing mix of renewable energy into the grid makes forecasting less predictable.

Two decades ago, Europe was at a similar market evolution point to India today with an emergent competitive market and a policy focus towards increasing sustainability investment. Europe has had to address the challenges provided by renewable capacity being added to the grid and to manage the transition, with forecasting now moving inexorably towards ‘big data’ and artificial intelligence as increasingly granular weather data is required to accommodate the growth in wind and solar generation.

Today, twenty-eight years on from the start of European energy deregulation, the number of weather data variables, their frequency of publication, and their data granularity has risen by several orders of magnitude. This was not a problem when weather data was updated every two hours, but the advent of real-time weather data streaming is putting the IT hardware performing the calculations under real pressure. Some may have suspected that the law of diminishing returns would set in, but this has been offset by the ability to create more complex AI-based models which engender greater accuracy.

This leads to data managers asking key questions, such as, “At what point does energy forecasting enter the realms of big data?” The evolution of energy forecasting from the 1990s to 2018 has resulted in a four or five order of magnitude increase in data quantities, assuming a move from deemed profiling to multi-channel smart meters, or from half-hourly thermal generation to 5-minute individual wind turbines of variable height, and solar panels under variable shade. As a rule of thumb, a commercial relational database starts to struggle badly if it has to process 500 million active records. A modest UK vertically integrated supplier-wind generator can generate millions of active rows per day, with each one being required at some point in machine learning, the precursor to AI. Thus big data can arrive very quickly where extensive renewable generation and smart meters are deployed.

A veteran of energy forecasting once quipped, “With big data I am vastly better informed, but absolutely none the wiser”. There is some merit in this statement, but the point of using AI in energy forecasting is that hitherto unsuspected patterns in the data from new combinations of extraneous variable suddenly become visible. This enables more innovative supply and generation tariffs to be created, and better rewards for flexible load and generation. The Holy Grail is to be able to tap into sources of flexible domestic load such as freezers and electric cars, combining them with local generation to create a mosaic of small, smart, and decentralised local grids, requiring minimal stabilisation from centralised thermal generation.

Assuming a European evolutionary timeline, for the purposes of determining energy forecasting data management requirements, fast-developing Asian markets will need to establish the point at which it intends to join the timeline – at full competition or partial, mixed metering or full big data, explicit rewards for flexibility, wholesale market-sized battery storage, smart meters and smart grids, etc.

The closer a market is to implementing smart meters and extensive renewable generation, the more likely it will be that they need to start their energy forecasting with a big data solution.

India is now at the point where it needs to make some decisions to support its changing generation market from a forecasting and data perspective. It needs to determine the most cost-effective balance of weather data, hardware capability, and forecasting accuracy, whether energy forecasting should be local or cloud-based, and whether Europe provides a model for its future data and forecasting.

The decisions made will not only help shape India’s market development, it may also provide a model for other developing Asian energy markets.

Jeremy Wilcox is managing director of the Energy Partnership, an independent Thailand-based energy and environment consulting firm.