| --- |
| tags: |
| - reinforcement-learning |
| - game-theory |
| - codenames |
| - neurips-2025 |
| - graph-neural-networks |
| - preference-learning |
| - llm-distillation |
| license: mit |
| --- |
| |
| # Codenames: Graph-Based RL with LLM-Guided Preference Distillation |
|
|
|  |
|  |
|  |
|
|
| This repository contains trained **Codenames agents** developed for the **NeurIPS 2025 MindGames Workshop**. |
| The system combines a structured graph-based reinforcement learning policy with **LLM-guided preference learning and distillation**, targeting improved risk calibration and decision robustness. |
|
|
| --- |
|
|
| ## Overview |
|
|
| The approach integrates: |
|
|
| - **Graph Neural Networks** for structured board and history representation |
| - **Proximal Policy Optimization (PPO)** for policy learning |
| - **Role-conditioned decoding** for spymaster and operative behaviors |
| - **Rollout-grounded preference learning** using large language models |
| - **Supervised fine tuning (SFT)** and **Direct Preference Optimization (DPO)** for teacher alignment |
| - **Knowledge distillation** from the aligned teacher back into a compact policy |
|
|
| The objective is to improve strategic consistency and reduce catastrophic failures such as assassin selections, while maintaining efficient inference suitable for interactive play. |
|
|
| --- |
|
|
| ## Game Configuration |
|
|
| - **Game**: Codenames |
| - **Board size**: 25 words |
| - **Roles**: Spymaster and Operative |
| - **Evaluation games**: 600 full episodes |
| - **Opponents**: Scripted baseline agents |
|
|
| --- |
|
|
| ## Policy Architecture |
|
|
| ### Graph-Based State Encoder |
| - Heterogeneous graph with **30–40 nodes** |
| - Node types include: |
| - Word nodes with semantic and state features |
| - Historical clue nodes |
| - Global summary node |
| - Node feature dimension: **35** |
| - Encoder: |
| - 3 Graph Attention layers |
| - 6 attention heads |
| - Hidden size 192 |
|
|
| ### Role Conditioning |
| - Shared policy trunk |
| - Role-conditioned action decoding: |
| - Clue generation and constraint handling for spymaster |
| - Guess selection and stopping decisions for operative |
|
|
| ### Model Size |
| - Total parameters: **~6.8M** |
| - Enables fast inference under competitive constraints |
|
|
| --- |
|
|
| ## Training Pipeline |
|
|
| Training follows a multi-stage curriculum: |
|
|
| 1. **Graph PPO Pretraining** |
| - PPO with clip ratio 0.2 |
| - Discount factor γ = 0.99 |
| - GAE λ = 0.95 |
| - Trained against scripted Codenames agents |
|
|
| 2. **Preference Generation via Rollouts** |
| - ~800 intermediate states sampled |
| - Candidate actions proposed by: |
| - Llama 3.1 Instruct |
| - Qwen 2.5 Instruct |
| - Each proposal evaluated using multiple stochastic rollouts |
| - Higher-return actions labeled preferred |
|
|
| 3. **Teacher Alignment** |
| - Supervised Fine Tuning on chosen actions |
| - Direct Preference Optimization using frozen reference model |
|
|
| 4. **Policy Distillation** |
| - Aligned teacher generates state-and-role to action labels |
| - Graph policy trained via cross-entropy imitation |
|
|
| 5. **PPO Refinement** |
| - PPO resumes using environment rewards |
| - Stabilizes policy after distillation |
|
|
| --- |
|
|
| ## Evaluation Results |
|
|
| Evaluation uses **600 full games** against scripted opponents. |
|
|
| | Agent | Win Rate | Assassin Rate | |
| |------|---------|---------------| |
| | Graph PPO | 44.8% | 12.6% | |
| | PPO + Distillation | 52.9% | 6.9% | |
|
|
| - Distillation yields an **8.1 point** absolute win-rate improvement |
| - Assassin-triggered losses are reduced by **45%** |
| - Improvements arise primarily from **better risk calibration**, not increased guessing aggressiveness |
|
|
| --- |
|
|
| ## Repository Contents |
|
|
| ### Policy Checkpoints |
| - `policy_models/policy_after_ppo.pt` |
| - `policy_models/policy_after_distill.pt` |
|
|
| ### Teacher Models |
| - `sft_model/` – supervised fine-tuned teacher |
| - `dpo_model/` – preference-aligned teacher |
|
|
| ### Configuration and Logs |
| - `master_config.json` |
| - `evaluation_results.json` |
|
|
| --- |
|
|
| ## Usage |
|
|
| ### Load Policy |
|
|
| ```python |
| import torch |
| from policy import GraphPolicy |
| |
| policy = GraphPolicy(...) |
| policy.load_state_dict(torch.load("policy_models/policy_after_distill.pt")) |
| policy.eval() |
| ``` |
|
|
| ### Loading Fine-tuned LLM |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| # Load SFT or DPO model |
| tokenizer = AutoTokenizer.from_pretrained("./sft_model") |
| model = AutoModelForCausalLM.from_pretrained("./sft_model") |
| |
| # Use for inference |
| inputs = tokenizer(prompt, return_tensors="pt") |
| outputs = model.generate(**inputs, max_new_tokens=32) |
| ``` |
|
|
| ## 🎓 Research Context |
|
|
| This work targets the **NeurIPS 2025 MindGames Workshop** with a focus on: |
|
|
| - Language models provide useful strategic priors when grounded by rollouts |
| - Graph-based representations enable structured reasoning in semantic games |
| - Distillation transfers high-level reasoning into efficient, deployable agents |
|
|
| ### Key Innovations |
|
|
| 1. **Heterogeneous Graph Representation**: Novel graph structure for Blotto game states |
| 2. **Ground-truth Counterfactual Learning**: Exploiting game determinism |
| 3. **Multi-scale Representation**: Field-level, round-level, and game-level embeddings |
| 4. **LLM-to-RL Distillation**: Transferring strategic reasoning to efficient policies |
|
|
|
|
| ## 📄 License |
|
|
| MIT License - See LICENSE file for details |
|
|
| ## 🙏 Acknowledgments |
|
|
| - Built for **NeurIPS 2025 MindGames Workshop** |
| - Uses PyTorch, HuggingFace Transformers, and PEFT |
| - Training infrastructure: NVIDIA H200 GPU |
|
|
| --- |
|
|
| **Generated**: {datetime.now().strftime("%Y-%m-%d %H:%M:%S")} |
| **Uploaded from**: Notebook Environment |
|
|
