[RL/Basic]MDP

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks

马尔可夫决策过程#

$$J=\mathbb [\sum_{t=0}^{\infty} \gamma^t r_t]$$

[RL/Perference-based] DPO

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Policy Optimization

DPO(Directed Preference Optimization)是一种不用训练奖励模型的对齐方法，用来让模型直接学习人类偏好，让模型更倾向于输出人类更喜欢的答案。

[RL/Policy Gradient] Actor-Critic

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Policy Gradient

引入基线$V(s)$降低方差。

[RL/Policy Optimization] PPO

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Policy Optimization

符号定义#

符号含义
$\pi_\theta(a|s)$参数为$\theta$的策略概率
$\theta_{old}$用于采样时的旧策略
$r_t(\theta)$概率比率，借鉴重要性采样的思想
$A_t$优势函数，衡量某动作相对于baseline的好坏
$\epsilon$截断阈值

目标函数#

定义单步的伪目标：

[RL/Value-based]Double Q-Learning

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Value-Based RL

减少过估计偏差

[RL/Value-based]Q-Learning

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Value-Based RL

Off Policy,学习最优$Q$

[RL/Value-based]SARSA

Reinforcement Learning
强化学习(Reinforcement Learning), Building Blocks, Value-Based RL

On Policy,遵循当前策略更新：