Multi-Turn Reinforcement Learning for Large Language Models: From Theory to Practice with Amazon SageMaker AI

Overview

Production LLM-based systems such as coding agents, web navigators, and tool-calling assistants operate over multiple turns of interaction with users, tools, and environments. This hands-on tutorial provides both a rigorous algorithmic and practical introduction to multi-turn RL finetuning for LLMs. Using Amazon SageMaker AI, participants progress through four labs: (1) environment and reward function design, (2) multi-turn trajectory collection and GRPO-based training, (3) reward densification and credit assignment strategies, and (4) evaluation, failure diagnosis and deployment.

Tutors

All authors are part of the Amazon team that launched Reinforcement Fine-Tuning (RFT) as a service on Amazon SageMaker AI and Amazon Bedrock, enabling developers to customize leading open-source models using techniques like RFT, Supervised Fine-Tuning (SFT), and Direct Preference Optimization (DPO).

Tutorial Outline

• Part 1 Motivation & Foundations 45 min

  Overview of multi-turn agentic challenges and why single-turn training fails. Multi-turn MDP formulation, GRPO algorithm introduction, reward taxonomy (trajectory-level, turn-level, composite), and environment design patterns.

  Lab: Implement a multi-turn tool-calling environment and composite reward function
• Part 2 Multi-Turn GRPO Training 30 min

  Configure rollout parameters, launch a SageMaker GRPO training job, inspect collected trajectories, monitor training dynamics via MLflow, and compare base model vs. RL-trained model on held-out tasks.

  Lab: End-to-end GRPO training on SageMaker AI
• Part 3 Reward Densification & Credit Assignment 50 min

  Sparse reward problem, densification strategies (milestone decomposition, LLM-as-judge), credit assignment (turn-level advantage estimation, temporal discounting), and reward hacking detection.

  Lab: Sparse vs. dense rewards, potential-based shaping, reward hacking experiments
• Part 4 Evaluation, Deployment & Wrap-Up 30 min

  Trajectory-level evaluation metrics, failure diagnosis via trajectory visualization, MLflow dashboards for stability warnings. One-click deployment to Amazon Bedrock. Future directions and Q&A.

  Lab: Evaluate, diagnose, and deploy trained agents

Target Audience

ML engineers, data scientists, applied researchers, and technical leaders who build LLM-powered multi-turn agentic systems (coding assistants, conversational agents, tool-using systems, web automation). Basic Python and Jupyter notebook familiarity and a foundational understanding of deep learning and LLM architectures (transformers, tokenization) are the only prerequisites. No prior experience with reinforcement learning or SageMaker is required.

Materials

All notebooks, environment code, reward function templates, slides, and demo videos will be provided via a public GitHub repository. Materials will be posted here closer to the tutorial date.