Reinforcement Learning for Recommendations: Optimizing User Satisfaction Across a Session
VK
Researchers from VK Research present a paper accepted at KDD 2026 workshop. They use reinforcement learning to optimize long-term user satisfaction in recommendation systems, addressing the mismatch between immediate engagement and long-term product metrics. The model, trained with REINFORCE and multi-step off-policy correction, aims to improve session-level optimization.
Artem Matveev from VK Research discusses their work accepted at the End-to-End Customer Journey Optimization workshop of KDD 2026. The paper, 'Session-Level Optimization for Large-Scale Retrieval using REINFORCE with Multi-Step Off-Policy Correction', addresses the problem that recommendation systems typically optimize for immediate user engagement (e.g., clicks or likes) but product metrics are long-term (e.g., retention or total time). They frame the recommendation problem as a Markov Decision Process (MDP) where the agent is the recommender, the environment is the user, states are user history, actions are recommended items, and rewards are user feedback. To learn a policy that maximizes long-term reward, they use policy gradient methods, specifically REINFORCE. Since on-policy learning is impractical, they employ off-policy learning with importance sampling to correct for the difference between the behavior policy (which generated the logs) and the target policy. They also discuss off-policy evaluation methods: Direct Method (DM), Inverse Propensity Scoring (IPS), and Doubly Robust (DR), and mention the challenge of high variance in importance weights.
- Abbreviations
- KDD = Knowledge Discovery and Data Mining — Knowledge Discovery and Data Mining
- RL = Reinforcement Learning — обучение с подкреплением
- MDP = Markov Decision Process — марковский процесс принятия решений
- OPE = Off-Policy Evaluation — оценка вне политики
- DM = Direct Method — прямой метод
- IPS = Inverse Propensity Scoring — оценка по обратной склонности
- DR = Doubly Robust — двойная устойчивая оценка
Source: Habr — хаб ИИ —
original