UNIPO: Unified Interactive Visual Explanation for RL Fine-Tuning Policy Optimization

Reinforcement learning has emerged as a dominant technique for fine-tuning the behavior of large language models, with policy optimization (PO) algorithms such as GRPO, DAPO, and Dr. GRPO emerging in rapid succession to advance state-of-the-art reasoning and alignment performance. However, the modular differences between these algorithms, including targeted improvements to clipping, advantage estimation, and reward aggregation, are introduced across separate papers with inconsistent notation, making them difficult to compare and intimidating to the non-expert community. We present UNIPO, the first interactive visualization tool that exposes the token-level training dynamics of RL fine-tuning algorithms through a unified design. UNIPO connects three complementary views — a high-level training overview, a step-level prompt and response inspector, and a side-by-side algorithm comparison — allowing learners to observe how individual design decisions propagate through training. Through two usage scenarios, we demonstrate how UNIPO supports both classroom instruction for non-experts and algorithm selection for AI practitioners.

Aeree Cho, Alexander D. Greenhalgh, Jonathan Bodea, Anthony Peng, Duen Horng (Polo) Chau

Turning the Tables: Empowering LLMs to Counter Deceptive Opponents

Large language models (LLMs) are increasingly deployed in negotiation settings where strategically motivated or deceptive behavior can have significant real-world consequences. While prior work investigates how to reduce deceptive tendencies in LLMs themselves, far less is known about how these systems respond when targeted by deception. In this paper, we study dialogue between a deceptive agent and a naive agent. Specifically, we construct a taxonomy of 20 deception strategies and evaluate their impact on the naive agent across three multi-turn negotiation domains. We find that the deceptive agent consistently reduces the utility of the naive agent, even when deception involves subtle misdirection rather than explicit falsehoods. We analyze the reasoning traces of the naive agent and find that LLMs rarely identify manipulative tactics, failing to challenge suspicious claims or reason about adversarial incentives. To counter this vulnerability, we introduce an in-context approach that induces deception-aware reasoning, enabling agents to probe inconsistencies and resist manipulation. Across all scenarios, this defense restores significant utility losses, building a stronger defense against deceptive behavior in real-world settings.

Jonathan Bodea, Marwa Abdulhai