[RL] Subgoal Automa

Reinforcement Learning
强化学习(Reinforcement Learning), Paper Reading
Induction and Exploitation of Subgoal Automata for Reinforcement Learning

简介#

  • 通过交织进行子目标自动机学习与强化学习
  • 让agent在与环境交互中自动发现任务的子目标结构，并利用该结构加速策略学习
  • 从交互轨迹中归纳子目标自动机，再用自动机指导强化学习，同时用新交互修正自动机

子目标概念机 Subgoal Automata#

子目标自动机是一种有限状态自动机，用于编码任务的子目标结构，其定义为tuple A=< $U,O,\delta_\phi,u_0,u_A,u_R$ >，关键组件包括

[RL/Basic]MDP

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks

马尔可夫决策过程#

$$J=\mathbb [\sum_{t=0}^{\infty} \gamma^t r_t]$$

[RL/Perference-based] DPO

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Policy Optimization

DPO(Directed Preference Optimization)是一种不用训练奖励模型的对齐方法，用来让模型直接学习人类偏好，让模型更倾向于输出人类更喜欢的答案。

[RL/Policy Gradient] Actor-Critic

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Policy Gradient

引入基线$V(s)$降低方差。

[RL/Policy Optimization] PPO

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Policy Optimization

符号定义#

符号含义
$\pi_\theta(a|s)$参数为$\theta$的策略概率
$\theta_{old}$用于采样时的旧策略
$r_t(\theta)$概率比率，借鉴重要性采样的思想
$A_t$优势函数，衡量某动作相对于baseline的好坏
$\epsilon$截断阈值

目标函数#

定义单步的伪目标：

[RL/Value-based]Double Q-Learning

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Value-Based RL

减少过估计偏差

[RL/Value-based]Q-Learning

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Value-Based RL

Off Policy,学习最优$Q$

[RL/Value-based]SARSA

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Value-Based RL

On Policy,遵循当前策略更新：