Counterfactually-Guided Causal Reinforcement Learning with Reward Machines

In causal reinforcement learning (RL), counterfactual reasoning deals with 'what if' situations and allows for investigating the potential consequences of actions or events that did not actually happen. In this paper, we combine counterfactual reasoning and reinforcement learning (RL) and propose Counterfactually-Guided Causal Reinforcement Learning with Reward Machines (CGC-RL). In CGC-RL, using observational data, we first compute the optimal counterfactual sequence with the highest probability of completing a given task. Then, we construct an RM compatible with the counterfactual sequence. We use the constructed RM to apply dynamic potential-based reward shaping to encourage the agent to follow the counterfactual sequence. We prove the policy-invariance under dynamic reward shaping with RMs. Finally, we implement CGC-RL in one case study and compare the results with three baselines. Our results show that CGC-RL outperforms the baselines.