A Deep Reinforcement Learning Approach for Managing Carbon Storage Reservoirs

Abstract:

Model-based optimization has played a central role in energy system design and optimal management. The complexity and high-dimensionality of many process-level models, especially those used for geosystem energy exploration and utilization, often lead to formidable computational costs when the dimension of decision space is also large. This work adopts elements of recently advanced deep learning techniques to solve a sequential decision-making problem in applied geosystem management. Specifically, a deep reinforcement learning framework was formed for optimal multi-period planning, in which a deep Q-learning network (DQN) agent was trained to maximize rewards by learning from high-dimensional inputs and from exploitation of its past experiences. To expedite computation, a deep, multitask autoregressive learning model was formed to approximate the high-dimensional, multistate transition functions. Both DQN and deep multitask learning are pattern based. For demonstration, the framework was applied to a spatially heterogeneous carbon sequestration reservoir. Carbon capture and storage is a geoengineering measure to mitigate climate change effect and is attracting broad interests because of the recent initiatives (e.g., 45Q tax credit in the U.S., and net-zero emissions). Two commonly used management strategies are optimized in this work, monitoring only strategy and brine extraction strategy. Both strategies are designed to mitigate potential risks due to pressure buildup while allowing the operator to maximize potential rewards. Results show that the DQN agent can identify the optimal policies to maximize the reward for given risk and cost constraints. Experiments also show that knowledge the agent gained from interacting with one environment is largely preserved when deploying the same agent in other similar environments, which is important for leveraging trained agents.