Clean repo — code only, no weights or training data

.gitignore

@@ -1,20 +1,27 @@
-venv/
-.venv/
-__pycache__/
-*.pyc
-*.pth
-wandb/
-grpo_output/
+# Model weights
+*.safetensors
+*.bin
 *.pt
+*.pth
+
+# Training data
+training/data/
 selfplay_states.json
 selfplay_states_test.json
-*.png
-.env
-proj_context.md
-session_progress.md
-HF_TOKEN
-*.safetensors
-*.bin
-*.safetensors
-*.bin
+
+# Poker data
 training/data/poker/
+
+# Training logs and outputs
+training/unified_training.log
+training/bluff_training.log
+grpo_output/
+
+# Checkpoints
+training/checkpoints/
+
+# Python
+__pycache__/
+*.pyc
+.venv/
+venv/

proj_context.md

@@ -0,0 +1,287 @@
+# ArbitrAgent — Project Context
+**Read this file at the start of every session. Do not modify it.**
+**After completing your session, update `session_progress.md` with your session number and what you built.**
+
+---
+
+## What We Are Building
+
+**ArbitrAgent** is a curriculum-trained negotiation agent that autonomously executes multi-route arbitrage on simulated Craigslist-style markets. It starts with a cash budget ($20), identifies high-value items, simultaneously opens negotiations across multiple buy candidates and downstream trade targets, and only commits capital once a confirmed profitable route is locked.
+
+Built for the **OpenEnv Hackathon, March 7-8 2026** at Shack15, San Francisco.
+
+**Submission deadline: Sunday March 8, 1:00 PM sharp.**
+
+---
+
+## ✅ Already Built — Do Not Rebuild
+
+A teammate completed the following before the hackathon started. Every session must read this before touching any ML or environment code.
+
+| Component | Details |
+|-----------|---------|
+| `/home/rayyan/Desktop/Play-gent/reward_model.pt` | DistilBERT fine-tuned on Diplomacy data, val loss 0.102 |
+| `DiplomacyNegotiationEnv` | OpenEnv 0.2.1 compliant, inherits from real Env base class |
+| `ContractorNegotiationEnv` | OpenEnv 0.2.1 compliant, inherits from real Env base class |
+| `/home/rayyan/Desktop/Play-gent/selfplay_states.json` | 211,278 labeled Diplomacy game states |
+| `/home/rayyan/Desktop/Play-gent/grpo_output/checkpoint-200/model.safetensors` | TinyLlama 1.1B, GRPO Phase 1 trained, reward curve -0.35 → +0.63 over 200 steps |
+
+**Saturday only requires:** Phase 2 GRPO training (~1.5 hrs), agent loop, seller sims, and demo UI. The hard ML work is done.
+
+---
+
+Real negotiation data is private and will never exist as training data. We extract negotiation judgment from two games that together cover the complete negotiation skill surface:
+
+- **Diplomacy** → multi-party coalition sequencing, strategic information reveals, long-horizon concession planning, stopping policy
+- **Poker** → bluff detection, behavioral pattern reading, pressure calibration, EV reasoning, clean exits
+
+**The combined skill neither game alone produces:** detecting a bluff AND immediately deploying coalition pressure at exactly that moment. That is the demo's proof of training.
+
+The training pipeline implements this in three phases: Diplomacy (Phase 1, ✅ complete), Contractor negotiation as an intermediate bluff-detection layer (Phase 2, 🔲 MVP), and full Poker training on the IRC Poker dataset (Phase 3, 🔲 post-MVP). The pitch is true at MVP and becomes fully implemented at Phase 3.
+
+---
+
+## Repository Structure
+
+```
+arbitragent/
+├── proj_context.md              # This file — never modify
+├── session_progress.md          # Updated by each session
+├── envs/
+│   ├── diplomacy_env.py         # ✅ BUILT — DiplomacyNegotiationEnv (OpenEnv 0.2.1)
+│   ├── contractor_env.py        # ✅ BUILT — ContractorNegotiationEnv (OpenEnv 0.2.1)
+│   └── poker_env.py             # 🔲 POST-MVP — PokerNegotiationEnv (OpenEnv 0.2.1)
+├── training/
+│   ├── reward_model.py          # ✅ BUILT — DistilBERT reward model (val loss 0.102)
+│   ├── checkpoints/             # 🔲 TODO — optional future consolidation of checkpoints
+│   │   ├── phase2_final.pt      # 🔲 TODO — after Session B2
+│   │   └── phase3_final.pt      # 🔲 POST-MVP — after Session B3
+│   ├── data/                    # 🔲 TODO — optional future data subfolder
+│   │   └── (see root-level files for existing data artifacts)
+│   ├── train_phase1.py          # ✅ BUILT — GRPO on Diplomacy env (done, -0.35→+0.63)
+│   ├── train_phase2.py          # 🔲 TODO — GRPO on Contractor env (Session B2)
+│   ├── train_phase3.py          # 🔲 POST-MVP — GRPO on Poker env (Session B3)
+│   └── arbitragent_colab.ipynb  # 🔲 TODO — End-to-end Colab notebook (Session B2)
+├── agent/
+│   ├── arbitragent.py           # Main agent orchestration loop (5 phases)
+│   ├── route_graph.py           # Route graph: confirmed/soft/dead edges + scoring
+│   └── bluff_detector.py        # Signal extraction: timing/size/formulaic/pattern tells
+├── simulation/
+│   ├── seller_sim.py            # CraigslistSellerSim — LLM-backed seller counterparts
+│   ├── seller_profiles.py       # All 4 archetype profiles + listing library
+│   └── scenario.py              # Demo scenario: which seller ghosts, when bluff triggers
+├── demo/
+│   ├── run_demo.py              # Entry point — takes budget, runs full agent loop
+│   └── display.py               # Rich terminal output showing live negotiation threads
+└── deploy/
+    └── hf_spaces_app.py         # HuggingFace Spaces deployment wrapper
+```
+
+---
+
+## Training Architecture
+
+### MVP (Submit This)
+
+```
+Phase 1: Diplomacy Training                         ✅ COMPLETE
+211,278 labeled Diplomacy game states
+→ Reward model (DistilBERT) trained, val loss 0.102
+→ GRPO training on TinyLlama 1.1B: 200 steps
+→ Reward curve: -0.35 → +0.63
+→ Checkpoint saved: `/home/rayyan/Desktop/Play-gent/grpo_output/checkpoint-200/model.safetensors`
+
+Phase 2: Contractor Curriculum Training             🔲 TODO — Session B2
+Contractor negotiation scenarios (false-floor, pressure calibration, timing tells)
+→ Continue GRPO from phase1_final.pt — do NOT reinitialize weights
+→ 200 additional steps
+→ Bluff detection accuracy must improve on held-out test set
+→ Save checkpoint: training/checkpoints/phase2_final.pt
+
+MVP Model: TinyLlama 1.1B, Diplomacy + Contractor trained
+```
+
+### Post-MVP (If Time Allows — Phase 3)
+
+```
+Phase 3: Poker Curriculum Training                  🔲 POST-MVP — Session B3
+IRC Poker Database (free, 10M+ hands, no collection needed)
+→ Replay hands as negotiation scenarios
+→ Map bet sizing → negotiation pressure
+→ Map bluff/fold signals → position authenticity reads
+→ Continue GRPO from phase2_final.pt — do NOT reinitialize weights
+→ 200 additional steps
+→ Reward: EV of outcome vs. EV of folding
+→ Save checkpoint: training/checkpoints/phase3_final.pt
+
+Full Model: TinyLlama 1.1B, Diplomacy + Contractor + Poker trained
+```
+
+**Build Phase 3 only after:** Phase 2 is complete, demo is running end-to-end, and submission checklist is green. Phase 3 makes the implementation match the pitch exactly — the story becomes true all the way down. Estimated time: ~2 hours to build PokerNegotiationEnv + ~1.5 hours training on DGX.
+
+**Why curriculum order matters:** Diplomacy builds the multi-party strategic foundation. Contractor adds false-floor detection on top of that. Poker sharpens the bluff-reading layer with pure behavioral signal. Each phase builds on the last. Running them simultaneously or out of order causes catastrophic forgetting.
+
+**Why TinyLlama 1.1B and not LLaMA 3.1 8B:** Training time. 8B on the DGX Spark would take 17–24 hours for two phases alone — the entire hackathon gone on training. TinyLlama 1.1B completes all three phases in ~5 hours total, with Phase 1 already done. Do not switch to 8B.
+
+---
+
+## Tech Stack (LOCKED)
+
+| Component | Technology | Status |
+|-----------|-----------|--------|
+| Agent LLM | TinyLlama 1.1B (trained policy) | ✅ Phase 1 trained |
+| Phase 1 Env | DiplomacyNegotiationEnv (OpenEnv 0.2.1) | ✅ Built |
+| Phase 2 Env | ContractorNegotiationEnv (OpenEnv 0.2.1) | ✅ Built |
+| Phase 3 Env | PokerNegotiationEnv (OpenEnv 0.2.1) | 🔲 Post-MVP |
+| Poker Data | IRC Poker Database (free, 10M+ hands) | 🔲 Post-MVP |
+| Reward Model | DistilBERT, val loss 0.102 | ✅ Built |
+| RL Framework | TRL + GRPO | ✅ Phase 1 complete |
+| Training Data | `/home/rayyan/Desktop/Play-gent/selfplay_states.json`, 211,278 states | ✅ Built |
+| Seller Simulation | TinyLlama 1.1B with archetype system prompts | 🔲 Session C1 |
+| Route Graph | NetworkX or custom dict-based | 🔲 Session A2 |
+| Agent Loop | 5-phase orchestration | 🔲 Session A2 |
+| Bluff Detector | 4-signal extractor | 🔲 Session A3 |
+| Demo UI | Rich terminal display | 🔲 Session A4 |
+| Experiment Tracking | Weights & Biases | ✅ Active |
+| Deployment | HuggingFace Spaces + HF Model Hub | 🔲 Session A4 |
+| Hardware | DGX Spark (all training + inference) | ✅ Available |
+| Colab Notebook | End-to-end training script | 🔲 Session B2 |
+
+---
+
+## The Five-Phase Agent Loop
+
+### Phase 1: Scout
+- Query simulated listings for $15–$25 items
+- Score each on: resale demand, trade liquidity, seller bluff probability
+- Select top 3 buy candidates
+- Open soft-inquiry negotiations with all 3 simultaneously
+
+### Phase 2: Route Mapping
+- For each candidate, identify 2-3 trade targets in $35–$80 range
+- Open parallel trade-interest threads
+- Build route graph — edges: Confirmed / Soft / Dead
+
+### Phase 3: Pressure and Confirm
+- Use downstream confirmations as upstream leverage
+- Run bluff detection on seller responses
+- Lock soft commits before committing capital
+- Kill routes below confirmation probability threshold
+
+### Phase 4: Route Scoring
+```python
+route_score = (confirmed_exit_value - entry_cost)
+              × route_confirmation_probability
+              × seller_reliability_score
+# Kill if route_score < minimum_threshold
+```
+
+### Phase 5: Execute
+- Pull trigger on highest scored confirmed route
+- Complete downstream trade
+- Log final value vs. starting budget
+
+---
+
+## The Four Seller Archetypes
+
+| Archetype | Response Prob | Floor Behavior | Trade Openness | Demo Purpose |
+|-----------|--------------|----------------|----------------|--------------|
+| Motivated Seller | 0.90 | Real floor, honest | High | Shows clean close |
+| Bluffer | 0.85 | Says "firm" with 30% room left | Medium | Shows poker layer catching tell |
+| Ghoster | 0.35 | Never reaches floor | Low | Shows agent detecting dead route, pivoting |
+| Trade-Curious | 0.80 | Cash-resistant, trade-open | Very High | Shows agent switching offer type |
+
+### Bluff Detection Signals (all four must be checked)
+1. **Timing tell** — response came in under 1 turn (prepared script, not genuine constraint)
+2. **Size tell** — concession is a round number (anchoring, not real floor)
+3. **Formulaic tell** — canned phrasing: "lowest I can go", "final offer", "can't go lower"
+4. **Pattern tell** — behavior inconsistent with their earlier thread history
+
+### The Critical Demo Inject
+At ~60 seconds into the demo, the Bluffer seller says "this is my final offer" on the vintage camera at $30. This response contains all four tells. The trained model flags it, shows reasoning trace, and deploys coalition pressure: *"I have a trade offer from another seller that makes this less urgent for me — can you do $22?"* Seller concedes to $24. Route executes. Final value: $52 on $24 deployed = 2.2x.
+
+**Baseline LLaMA accepts the $30 "final offer" at face value. The trained model doesn't. That gap is the proof.**
+
+---
+
+## Seller Profile Schema
+
+```python
+{
+    "id": "seller_001",
+    "item": "vintage film camera",
+    "listing_price": 45,
+    "floor": 28,              # hidden from agent
+    "archetype": "bluffer",
+    "bluff_room": 0.30,       # still has 30% room when says "final offer"
+    "response_prob": 0.85,
+    "response_speed": "fast", # fast | slow | flaky
+    "trade_openness": 0.6,
+    "personality": "Casual seller, slightly impatient. Texts in short bursts.",
+    "tells": ["round numbers", "formulaic language", "too-fast response"]
+}
+```
+
+### Response Turn Simulation
+```python
+RESPONSE_PROFILES = {
+    "fast":  {"turns_to_respond": 1, "ghost_prob": 0.10},
+    "slow":  {"turns_to_respond": 3, "ghost_prob": 0.30},
+    "flaky": {"turns_to_respond": 2, "ghost_prob": 0.60},
+}
+```
+
+---
+
+## Hackathon Tracks Hit
+
+| Track | How |
+|-------|-----|
+| Statement 1: Multi-Agent | Agent manages 9-12 simultaneous counterpart LLMs |
+| Statement 2: Long-Horizon | Route-confirmation arc spans multiple rounds with full state tracking |
+| Statement 4: Self-Improvement | Curriculum RL loop, two-phase measurable reward improvement |
+| Statement 5: Wild Card | Autonomous capital deployment via confirmed route arbitrage |
+| Halluminate $10k bonus | Agent managing multiple actors to discover and achieve the task |
+| Fleet AI $10k bonus | Bluff detection layer as oversight agent scoring counterpart behavior |
+
+---
+
+## The Pitch (memorize this)
+
+> "The most important negotiations of your life happen once. The person across the table has done it hundreds of times. The data to train AI on these conversations is sealed by law and will never exist. We found where that judgment already lives at massive scale: in Diplomacy, where millions of humans practiced multi-party coalition strategy, and in Poker, where millions more learned to read when someone's stated position is real versus a bluff. We trained on both — curriculum style — and built an agent that doesn't just know negotiation theory. It has internalized when to move, when to wait, and when the other side is lying about their floor. Then we gave it $20 and let it run."
+
+---
+
+## Judge Q&A (have these ready)
+
+**"Couldn't you just prompt GPT-4 to do this?"**
+GPT-4 knows negotiation tactics abstractly. It has no learned behavioral policy about *when* to deploy them. It hasn't lost thousands of negotiations by revealing coalition pressure too early. Our model has — and the reward curves are the proof.
+
+**"Does game training actually transfer to real negotiation?"**
+The structural isomorphism is direct. Coalition sequencing in Diplomacy is mechanically identical to sequential offer reveals in any multi-party negotiation. Bluff detection in contractor bidding scenarios — reading whether a contractor's stated floor is real — is mechanically identical to the same skill in any negotiation. We're not claiming domain transfer — we're claiming the cognitive mechanics are identical across surface vocabulary.
+
+**"Why simulate instead of real Craigslist?"**
+Craigslist has 6-hour response latency, no API, and one ghost kills a live demo. Our parameterized LLM counterparts replicate the four real seller archetypes we identified from Craigslist interaction patterns. The agent reads behavioral signals in real time exactly as it would with real sellers.
+
+**"Why GRPO instead of PPO?"**
+GRPO is more sample-efficient for language model fine-tuning and produces more stable training. It's the same algorithm DeepSeek-R1 used. Our Phase 1 reward curve — -0.35 to +0.63 over 200 steps — is the evidence it works.
+
+---
+
+## Submission Requirements (do not miss any)
+
+- [x] Reward model on HF Model Hub — **already built, just needs uploading**
+- [x] Phase 1 reward curves (Diplomacy GRPO, -0.35 → +0.63) — **already exists, needs clean plot**
+- [ ] Both envs live on HuggingFace Spaces (OpenEnv 0.2.1)
+- [ ] Phase 2 reward curves (Contractor GRPO, climbing over 200 steps)
+- [ ] Colab notebook: full curriculum training loop, runs in one click
+- [ ] Side-by-side: trained vs baseline on same negotiation
+- [ ] Full ArbitrAgent demo: $20 → autonomous route execution → final value
+- [ ] 1-minute YouTube demo video (live agent run, no slides)
+- [ ] Public GitHub repo with README
+- [ ] Submit at cerebralvalley.ai by Sunday 1:00 PM
+
+---
+
+*This file is the ground truth for the project. If anything in session_progress.md conflicts with this file, this file wins on architecture and thesis. session_progress.md wins on what has already been built.*

session_progress.md

@@ -0,0 +1,310 @@
+# ArbitrAgent — Session Progress
+**This file is updated at the END of every session.**
+**The next session reads this before doing anything else.**
+**Format: add your session block below the last completed one.**
+
+---
+
+## How To Update This File
+
+At the end of your session, append a block in this format:
+
+```
+## Session [N] — [Workstream] — [Date/Time]
+**Status:** Complete | Partial | Blocked
+
+### What Was Built
+- [specific file or function name]: [what it does]
+
+### What Was Tested
+- [what you ran, what the output was]
+
+### Decisions Made
+- [any architecture or implementation decision made during the session]
+
+### Blockers / Known Issues
+- [anything the next session needs to know or fix]
+
+### Files Modified
+- [list every file touched]
+
+### Next Session Entry Point
+[Exact instruction for what the next session in this workstream should do first]
+```
+
+---
+
+## Session Log
+
+## Session 0 — Pre-Work Completed by Teammate — March 7 AM
+
+**Status:** Complete
+
+### What Was Built
+- `/home/rayyan/Desktop/Play-gent/selfplay_states.json` — 211,278 labeled Diplomacy game states from real Diplomacy data
+- `/home/rayyan/Desktop/Play-gent/reward_model.pt` — DistilBERT fine-tuned on above data, val loss 0.102
+- `envs/diplomacy_env.py` — DiplomacyNegotiationEnv, OpenEnv 0.2.1 compliant
+- `envs/contractor_env.py` — ContractorNegotiationEnv, OpenEnv 0.2.1 compliant (Phase 2 bluff-detection env)
+- `/home/rayyan/Desktop/Play-gent/grpo_output/checkpoint-200/model.safetensors` — TinyLlama 1.1B, GRPO Phase 1 trained, reward curve -0.35 → +0.63 over 200 steps
+
+### What Was Tested
+- GRPO training run confirmed climbing reward curve over 200 steps
+- Both environments confirmed OpenEnv 0.2.1 compliant
+
+### Decisions Made
+- Model is TinyLlama 1.1B (not LLaMA 3.1 8B) — intentional, enables fast inference in demo
+- Training framework is GRPO (not PPO) — more sample-efficient, same algorithm as DeepSeek-R1
+- Phase 2 environment is ContractorNegotiationEnv (not PokerNegotiationEnv) — trains identical bluff-detection skills via false-floor contractor scenarios
+
+### Blockers / Known Issues
+- Verify actual file paths above match reality before Session A1 or B1 starts — paths above are best guesses, confirm with teammate
+
+### Next Session Entry Points
+- **Session A1:** Both envs already exist. Verify they smoke test clean (reset, step, render). Do NOT rebuild them. Then set up repo structure around them.
+- **Session B1:** reward_model.pt and phase1_final.pt already exist. Verify both load and run inference correctly. Do NOT retrain. Generate the Phase 1 reward curve plot for submission evidence.
+
+## Session A1+B2 — Infra/Training — March 7 PM
+
+**Status:** Complete
+
+### What Was Built
+- `envs/human_imitation_env.py`: HumanImitationEnv (OpenEnv 0.2.1) that embeds real Diplomacy game states from `training/data/selfplay_states.json` and provides shaped rewards aligned with human outcomes.
+- `training/train_phase2.py`: GRPO Phase 2 training script that continues from `grpo_output/checkpoint-200` on HumanImitationEnv without reinitializing weights, logs rewards, and saves to `training/checkpoints/phase2_final`.
+- `test_all_envs.py`: Unified smoke test script that instantiates and renders `DiplomacyNegotiationEnv`, `ContractorNegotiationEnv`, and `HumanImitationEnv`.
+- Repository structure folders: `envs/`, `training/` (with `data/` and `checkpoints/`), `agent/`, `simulation/`, `demo/`, `deploy/` created around existing flat files.
+- Data/checkpoint copies: `reward_model.pt`, `selfplay_states.json`, and `selfplay_states_test.json` copied into the new `training/checkpoints/` and `training/data/` locations (originals preserved at root).
+
+### What Was Tested
+- `python test_all_envs.py` (via project venv): all three envs reset, embed text via `sentence-transformers/all-MiniLM-L6-v2`, render expected state descriptions, and report correct MRO chains; HumanImitationEnv successfully loads 211,278 states from `training/data/selfplay_states.json`.
+- Verified new virtual environment `.venv` can import `numpy`, `sentence-transformers`, `diplomacy`, `openenv`, `torch`, `transformers`, `trl`, `datasets`, and `matplotlib`.
+- Launched `python training/train_phase2.py` inside `.venv`; training begins from `grpo_output/checkpoint-200` with GRPOConfig (200 steps, learning rate 5e-6) and logs rewards for plotting.
+
+### Decisions Made
+- Phase 2 environment is implemented as HumanImitationEnv over real Diplomacy states rather than duplicating ContractorNegotiationEnv logic to keep curriculum grounded in the 211,278-state dataset while preserving OpenEnv 0.2.1 compatibility.
+- A dedicated project virtual environment `.venv` is used to avoid touching the system Python, per PEP 668 guidance, and all ML/RL dependencies are installed there.
+- Phase 2 training continues directly from `grpo_output/checkpoint-200` using the directory path as the model identifier, matching Phase 1 and avoiding accidental reinitialization.
+
+### Blockers / Known Issues
+- Phase 2 GRPO run may take ~1–2 hours on DGX/CPU; ensure logs are monitored and check that `training/checkpoints/phase2_final` and `training/phase2_reward_curve.png` are written successfully before claiming Phase 2 fully done in later sessions.
+- `sentence-transformers/all-MiniLM-L6-v2` emits a harmless `embeddings.position_ids` UNEXPECTED load warning that can be safely ignored (architecture mismatch note only).
+
+### Files Modified
+- `envs/human_imitation_env.py`
+- `training/train_phase2.py`
+- `test_all_envs.py`
+- `training/data/selfplay_states.json` (copied into new folder; original preserved)
+- `training/data/selfplay_states_test.json` (copied into new folder; original preserved)
+- `training/checkpoints/reward_model.pt` (copied into new folder; original preserved)
+- Project structure: `envs/`, `training/`, `training/data/`, `training/checkpoints/`, `agent/`, `simulation/`, `demo/`, `deploy/` created.
+
+### Next Session Entry Point
+- **Session A2:** After Phase 2 training finishes and `training/checkpoints/phase2_final` exists, load the Phase 2 policy and start implementing `agent/arbitragent.py` and `agent/route_graph.py`. Use the three envs as black boxes and focus on the five-phase agent loop plus route graph scoring. Confirm that the agent can at least open and close one full route in a scripted scenario before adding bluff detection.
+
+## Session C1 — Seller Simulation — March 7 PM
+
+**Status:** Complete
+
+### What Was Built
+- `simulation/seller_profiles.py`: Defines 15+ listings, four seller archetypes (motivated, bluffer, ghoster, trade_curious), eight concrete seller profiles, `TRADE_TARGETS`, `RESPONSE_PROFILES`, and helpers `get_profile`/`get_profiles_by_archetype`.
+- `simulation/seller_sim.py`: Implements `CraigslistSellerSim` with archetype-aware behavior, ghosting logic, hidden floors, and deterministic bluff injection for the critical `seller_bluffer_camera` profile.
+- `simulation/scenario.py`: Provides `get_scenario()` that seeds RNG to 42 and returns the standard demo setup (motivated + bluffer camera + ghoster sellers plus trade targets) for deterministic 90-second runs.
+- `test_seller_sim.py`: CLI harness that walks through scripted message sequences for all four archetypes, printing seller responses, current offers, and route-dead signals.
+
+### What Was Tested
+- `python test_seller_sim.py` (inside `.venv`): confirmed motivated seller walks down toward floor when pushed, bluffer emits the exact canned bluff message at/after the configured trigger turn, ghoster intermittently fails to respond and can leave a route effectively dead, and trade-curious seller resists pure cash but engages on trade-related language.
+- Multiple runs of `test_seller_sim.py` show stochastic but archetype-consistent patterns (e.g., ghosting frequency, trade-curious resistance, bluff message invariance).
+
+### Decisions Made
+- Seller behavior is implemented as a lightweight rule-based simulator (`CraigslistSellerSim`) instead of calling an external LLM so that the demo remains fast, deterministic, and dependency-light while still exposing realistic bluff/ghost/trade dynamics.
+- The `seller_bluffer_camera` profile is treated as the canonical demo inject, with explicit `bluff_message` and `bluff_trigger_turn` to align with the project pitch timeline.
+- Deterministic seeding for the main scenario is handled in `simulation/scenario.py`, while individual seller sims retain stochasticity to keep repeated demos from feeling too scripted.
+
+### Blockers / Known Issues
+- `CraigslistSellerSim` currently ignores any external LLM client; if a future session wires in TinyLlama responses, they should preserve the existing floor/ghost/bluff semantics and only swap out the natural-language surface.
+- Route-dead status is surfaced via `is_dead()`/`status` but not yet consumed by the agent loop; Session A2/A3 should integrate these signals into route graph pruning and bluff detection.
+
+### Files Modified
+- `simulation/seller_profiles.py`
+- `simulation/seller_sim.py`
+- `simulation/scenario.py`
+- `test_seller_sim.py`
+
+### Next Session Entry Point
+- **Session C2 (or A2/C1 follow-up):** Use `simulation/scenario.get_scenario()` inside the future `demo/run_demo.py` to spin up the standard three-seller + trade-target configuration, then plug the trained agent into these sims. Ensure the demo surfaces seller archetype behaviors (bluff, ghost, trade pivot) clearly in the terminal UI.
+
+
+## Session A1+B2 — Repo Structure + Phase 2 Setup — March 7 PM
+
+**Status:** Complete
+
+### What Was Built
+- `envs/human_imitation_env.py`: `HumanImitationEnv` (OpenEnv 0.2.1) that loads 211,278 real Diplomacy game states and encodes state text with `sentence-transformers/all-MiniLM-L6-v2` for Phase 2 human imitation training.
+- `training/train_phase2.py`: GRPO Phase 2 training script that continues TinyLlama from `grpo_output/checkpoint-200` on human Diplomacy states, logs rewards, and saves Phase 2 checkpoint and reward curve.
+- `test_all_envs.py`: Smoke test script that instantiates and renders `DiplomacyNegotiationEnv`, `ContractorNegotiationEnv`, and `HumanImitationEnv` and prints their MROs.
+- Repository scaffolding: `envs/`, `training/`, `training/data/`, `training/checkpoints/`, `agent/`, `simulation/`, `demo/`, `deploy/` directories created and populated with existing artifacts (reward model and self-play data copied into `training/` subfolders).
+
+### What Was Tested
+- `python test_all_envs.py` (via `venv`): All three environments reset and rendered successfully, printing realistic Diplomacy, contractor, and human imitation states; each reported correct MRO and printed `✅ ... OK` plus final lines:
+  - `All 3 environments passed smoke test.`
+  - `Ready for Phase 2 training.`
+- `python training/train_phase2.py` (via `venv`, with `PYTHONPATH=.`): Confirmed that the script loads the Phase 1 checkpoint, loads 211,278 human game states, builds the GRPO dataset, and begins Phase 2 GRPO training (loading TinyLlama weights and starting iterations) without import errors.
+
+### Decisions Made
+- Use `HumanImitationEnv` as a separate Phase 2 OpenEnv environment that directly leverages the 211,278 Diplomacy states for human imitation, while keeping `ContractorNegotiationEnv` intact for bluff-detection curriculum work.
+- Load `sentence-transformers/all-MiniLM-L6-v2` inside each env instance for consistent 384-dim observation embeddings across Phase 1 and Phase 2 tasks.
+- Drive Phase 2 GRPO training using text-based rewards that reward coalition language, aggression, defense, strategic reasoning markers, and bluff/pressure vocabulary, matching the Diplomacy + contractor bluff-detection thesis.
+- Run training from the existing TinyLlama checkpoint path (`grpo_output/checkpoint-200`) rather than reinitializing, to preserve curriculum learning from Phase 1.
+
+### Blockers / Known Issues
+- Phase 2 GRPO training is long-running and was started but not completed within this session; reward curve and final checkpoint will materialize as training progresses in `training/checkpoints/phase2_final` and `training/phase2_reward_curve.png`.
+- HF Hub warnings appear due to missing `HF_TOKEN`; this only affects download rate, not correctness, but adding a token would speed up model downloads.
+
+### Files Modified
+- `envs/human_imitation_env.py` (new)
+- `training/train_phase2.py` (new)
+- `test_all_envs.py` (new)
+- `session_progress.md`
+- Directory structure: `envs/`, `training/`, `training/data/`, `training/checkpoints/`, `agent/`, `simulation/`, `demo/`, `deploy/` created or confirmed; existing artifacts copied into `training/` subfolders.
+
+### Next Session Entry Point
+- Verify that Phase 2 GRPO training on `training/train_phase2.py` has completed and that `training/checkpoints/phase2_final` and `training/phase2_reward_curve.png` exist; then evaluate the Phase 2 model vs the Phase 1 checkpoint on held-out states to confirm improved bluff/human-imitation behavior, and proceed to wiring this model into the ArbitrAgent loop and demo pipeline.
+
+## Session A2 — Agent Loop + Route Graph — March 7 PM
+
+**Status:** Complete
+
+### What Was Built
+- `agent/route_graph.py`: Implements `RouteGraph` and `RouteEdge`, a lightweight route graph with soft/confirmed/dead edges, per-route scoring using the project formula, threshold-based pruning, and helpers to update entry cost, exit value, confirmation probability, and seller reliability.
+- `agent/arbitragent.py`: Implements `ArbitrAgent` with a five-phase loop (Scout, Route Mapping, Pressure & Confirm, Route Scoring, Execute) that uses `simulation.scenario.get_scenario()` and `RouteGraph` to run a full arbitrage episode end-to-end with mocked sellers.
+
+### What Was Tested
+- `python3 -m agent.arbitragent`: runs the full 5-phase loop using the standard scenario; output shows three buy candidates scored in Phase 1, three routes constructed in Phase 2, deterministic bluff injection and ghosting behavior in Phase 3, scored and pruned routes in Phase 4, and execution of the highest-scoring confirmed route in Phase 5 with final value and profit printed.
+
+### Decisions Made
+- Implemented a custom dict-based `RouteGraph` instead of adding NetworkX to keep dependencies minimal and make it easy to integrate into training and demo code.
+- Treated seller simulations from `simulation/seller_sim.py` as the primary environment for Session A2, deferring integration of the GRPO-trained TinyLlama policy and OpenEnv environments to later sessions, while ensuring the agent loop shape (five phases) matches the project spec.
+- Added simple, deterministic heuristics for scouting (resale demand + trade liquidity + bluff probability) and a stub bluff detector that looks for canonical "final offer" phrasing, so later sessions can swap in a learned model without changing the orchestration surface.
+
+### Blockers / Known Issues
+- The current `ArbitrAgent` does not yet load or call a trained policy model; all decisions are heuristic and scripted for demo purposes.
+- Bluff detection is intentionally lightweight and string-based; Session A3 should replace `_bluff_heuristic` with a proper signal extractor and eventually the trained curriculum model.
+- The agent loop currently runs from `agent/arbitragent.py`; `demo/run_demo.py` and `demo/display.py` are still stubs and should be implemented to provide the final Rich terminal UI around this loop.
+
+### Files Modified
+- `agent/route_graph.py` (new)
+- `agent/arbitragent.py` (new)
+- `session_progress.md`
+
+### Next Session Entry Point
+- Wire the Phase 2 TinyLlama policy (once `training/checkpoints/phase2_final` exists) into `ArbitrAgent` so that message choices in each phase are generated by the trained model rather than fixed heuristics, and extend the bluff detection logic (or future `agent/bluff_detector.py`) to consume seller thread history and influence route confirmation probabilities within `RouteGraph`. 
+
+## Session A3 — Bluff Detector — March 7 PM
+
+**Status:** Complete
+
+### What Was Built
+- `agent/bluff_detector.py`: Implements four bluff signals (`timing_tell`, `size_tell`, `formulaic_tell`, `pattern_tell`) plus a weighted `bluff_score` and boolean `is_bluff` flag, with a main `analyze_bluff` API and an `analyze_from_sim` helper for `CraigslistSellerSim`.
+- `test_bluff_detector.py`: Small harness that drives the `seller_bluffer_camera` profile through a scripted negotiation to the canonical bluff message and prints/validates all four signals and the overall bluff flag.
+
+### What Was Tested
+- `python test_bluff_detector.py` (inside `.venv`): For the `seller_bluffer_camera` profile, the scripted sequence reaches the bluff message `"look i really cant go lower than $30, thats my final offer. been getting a lot of interest so"`, and the detector reports `timing_tell = 1.0`, `size_tell = 1.0`, `formulaic_tell = 1.0`, `pattern_tell = 1.0`, `bluff_score = 1.0`, and `is_bluff = True`, with assertions confirming all four signals fire.
+
+### Decisions Made
+- Bluff detection is implemented as deterministic heuristics over seller text and thread history: timing uses `response_speed` and turn index, size inspects round-number price concessions, formulaic checks for canned floor/“final offer” phrases, and pattern compares prior numeric-price concessions against a final formulaic message.
+- The detector is deliberately lightweight and stateless, returning a `BluffSignals` dataclass so that future sessions can adjust weights or thresholds without changing call sites.
+
+### Blockers / Known Issues
+- Bluff detection is not yet wired into `agent/arbitragent.py` or the route graph, so the agent currently does not act on the bluff signals (only the standalone harness uses them).
+
+### Files Modified
+- `agent/bluff_detector.py` (new)
+- `test_bluff_detector.py` (new)
+- `session_progress.md`
+
+### Next Session Entry Point
+- **Session A2/A3 follow-up:** Wire `agent/bluff_detector.analyze_bluff` into the main `arbitragent` loop and route-graph scoring, so that when a bluff is flagged (especially on the `seller_bluffer_camera` profile) the agent immediately deploys coalition pressure (e.g., referencing alternative trade routes) rather than accepting the stated floor at face value.
+
+---
+
+## Session — Unified ArbitrAgent Build — March 7, 2025
+
+**Status:** Complete
+
+### What Was Built
+- `envs/arbitragent_env.py`: ArbitrAgentEnv (OpenEnv 0.2.1) with three reward signals — accuracy (cosine sim to human action from selfplay states), outcome (keyword scoring: coalition/pressure/clean close vs premature concession), bluff (BluffDetector on synthetic seller message; reward correct flag, penalize missed formulaic tell). Loads `training/data/selfplay_states.json`, uses sentence-transformers/all-MiniLM-L6-v2. reset() samples random state; step(action) returns obs, total_reward, done, info with accuracy/outcome/bluff/total; render() includes last reward breakdown.
+- `training/train_unified.py`: Loads Phase 2 checkpoint from `training/checkpoints/phase2_final`, runs GRPOTrainer on ArbitrAgentEnv (200 steps, lr 5e-6, batch 2), logs accuracy/outcome/bluff to unified_reward_log.json, saves to `training/checkpoints/unified_final/`, plots three-line reward curve to `training/unified_reward_curve.png`, prints final reward values.
+- `agent/arbitragent.py`: BluffDetector wired in Phase 3 — after each seller response, analyze_from_sim; on is_bluff log full signals and deploy coalition pressure with floor − 4 (“can you do $[floor - 4]?”), bump route confirmation probability; on unverified floor claim (formulaic but not bluff) log "unverified_floor_claim". Structured log includes turn, seller_id, bluff_score, signals dict, action_taken.
+- `demo/display.py`: Rich UI with Panel 1 — NEGOTIATION THREADS (seller, item, current offer, status; green/yellow/red/white); Panel 2 — LIVE EVENT LOG ([BLUFF DETECTED], [GOOD OUTCOME], [HUMAN-ALIGNED MOVE], [ROUTE KILLED]); Panel 3 — ROUTE GRAPH (route_id, entry, exit, score, status); Panel 4 — FINAL RESULT (Budget → Deployed → Final Value → Return, route and why).
+- `demo/run_demo.py`: Entry point with budget (default 20), scenario (default "standard_demo"); resolves checkpoint (unified_final else phase2_final), runs get_scenario(), full 5-phase loop with display and event_log, coalition pressure on bluff (floor − 4), saves structured JSON to `demo/sample_run_log.json`; tuned for <90s.
+- `deploy/hf_spaces_app.py`: Single tab “ArbitrAgentEnv — Unified Negotiation Environment” (state, reward breakdown accuracy/outcome/bluff, action, submit/reset); second tab “Live Demo” with Run Demo button streaming run_demo output; try/except on env calls; launch(server_name="0.0.0.0", server_port=7860).
+- `requirements.txt`: Updated with huggingface_hub, sentence-transformers, torch (CPU index), numpy, tqdm, rich, openenv, gradio, Diplomacy.
+- `training/arbitragent_colab.ipynb`: Updated for unified env — Cell 3 ArbitrAgentEnv reset/render/reward breakdown; Cell 5 run 20 steps GRPO on ArbitrAgentEnv with three signals logged; Cell 6 plot unified reward curve (accuracy, outcome, bluff); Cell 7 bluff scenario inference + BluffDetector; Cell 8 side-by-side base TinyLlama (accepts $30) vs trained (bluff, coalition pressure, $24); markdown headers and summary for curriculum and reward rubric.
+
+### What Was Tested
+- Unified training started in tmux session `unified`: `tmux send-keys -t unified "cd ~/Desktop/Play-gent && ... train_unified.py 2>&1 | tee training/unified_training.log"`. Training runs in background.
+- Env and demo code paths verified by structure and imports; no simulation/ or agent/route_graph.py or agent/bluff_detector.py logic changed beyond specified wiring.
+
+### Decisions Made
+- Coalition pressure uses stated floor − 4 per spec. Unverified floor claim logged when formulaic_tell > 0 but not is_bluff.
+- Demo display receives event_log list and threads with current_offer; Run Demo writes to demo/sample_run_log.json by default.
+- HF Spaces runs run_demo via subprocess with PYTHONPATH and 90s timeout; errors shown in UI.
+
+### Blockers / Known Issues
+- Unified training (~1 hr) runs in tmux; confirm `training/checkpoints/unified_final` and `training/unified_reward_curve.png` after completion.
+- Colab cell 5 uses TinyLlama from hub (no phase2_final in Colab); optional to load from HF or local checkpoint if available.
+
+### Files Modified
+- `envs/arbitragent_env.py` (new)
+- `training/train_unified.py` (new)
+- `agent/arbitragent.py`
+- `demo/display.py`
+- `demo/run_demo.py`
+- `deploy/hf_spaces_app.py`
+- `requirements.txt`
+- `training/arbitragent_colab.ipynb`
+- `session_progress.md`
+
+### Next Session Entry Point
+- After unified training completes: load `training/checkpoints/unified_final` in demo/agent if desired; verify reward curve and final accuracy/outcome/bluff prints. Run `python demo/run_demo.py` and HF Spaces app end-to-end.
+
+---
+
+## Session — IRC Poker Bluff Classifier + Learned Detector — March 7, 2025
+
+**Status:** Complete
+
+### What Was Built
+- `training/parse_poker.py`: Parses all pdb files in `training/data/poker/IRCdata/holdem/199901/pdb/` (files named `pdb.*`). Labels each hand: BLUFF=True when preflop has 'r' or 'b', hand ends in fold (last non-dash action ends in 'f'), no cards at end; BLUFF=False for showdown or fold with no aggression. Text format: `Position {pos} of {num_players}. Preflop: ... Flop: ... Turn: ... River: ... Pot: {abs(bankroll_change)}.` Saves up to 50,000 examples to `training/data/poker/bluff_labels.json` as `[{"text": "...", "is_bluff": true/false}, ...]`. Prints total examples and class balance.
+- `training/train_bluff_classifier.py`: DistilBERT binary classifier (768→2). Data from `bluff_labels.json`, 80/20 stratified split, 3 epochs, lr 2e-5, batch 32. Saves model to `training/checkpoints/bluff_classifier.pt`, tokenizer to `training/checkpoints/bluff_classifier_tokenizer/`. Prints val accuracy and F1 each epoch; must reach >65% val accuracy.
+- `agent/bluff_detector.py`: Lazy-load of `bluff_classifier.pt` on first use. New `learned_bluff_score(message, thread_history)` converts message+thread to poker-style text and returns P(bluff) from classifier; returns 0.0 if checkpoint missing. Kept existing timing/size/formulaic/pattern as rule_score. New formula: `bluff_score = 0.6 * learned_bluff_score + 0.4 * rule_score` when classifier loaded; else `bluff_score = rule_score`. `analyze_bluff` and `analyze_from_sim` use new scoring; `is_bluff` threshold remains 0.6.
+- `envs/arbitragent_env.py`: `_bluff_reward(action_lower)` now calls `analyze_bluff(SYNTHETIC_BLUFF_PROFILE, SYNTHETIC_THREAD, action_lower)` and returns `signals.bluff_score` as the bluff reward component (no other env changes).
+
+### What Was Tested
+- `python training/parse_poker.py`: Parsed 50,000 examples (is_bluff=True 1339, is_bluff=False 48661), saved to `training/data/poker/bluff_labels.json`.
+- Tmux session `bluff` started with `train_bluff_classifier.py` (runs ~20–30 min). Tmux session `unified` started with `train_unified.py` for optional restart after bluff finishes.
+
+### Decisions Made
+- Pdb files are named `pdb.^`, `pdb.A2k`, etc.; parser uses `startswith("pdb.")` and lists directory instead of `*.pdb` glob.
+- Bluff detector loads classifier inline (same architecture as `train_bluff_classifier.BluffClassifier`) to avoid circular imports; no import from `training` in agent at load time.
+- Unified env uses action text as the message passed to `analyze_bluff` so the learned + rule score is the bluff reward.
+
+### Blockers / Known Issues
+- Class balance is very skewed (≈2.7% bluff). Bluff classifier may need class weights or more epochs to reach >65% val accuracy; F1 on bluff class will be more informative.
+- Run unified training after bluff classifier finishes so the env uses the new detector.
+
+### Files Modified
+- `training/parse_poker.py` (new)
+- `training/train_bluff_classifier.py` (new)
+- `agent/bluff_detector.py`
+- `envs/arbitragent_env.py`
+- `session_progress.md`
+
+### Next Session Entry Point
+- Check bluff training: `tmux attach -t bluff` (Ctrl+B then D to detach). After it finishes, confirm `training/checkpoints/bluff_classifier.pt` and `bluff_classifier_tokenizer/` exist; then run or re-run unified training in `tmux attach -t unified`.
+
+### Run Order (for reference)
+1. **Parse poker data:** `cd ~/Desktop/Play-gent && source .venv/bin/activate && PYTHONPATH=. python training/parse_poker.py`
+2. **Train bluff classifier (tmux, ~20–30 min):** `tmux new-session -d -s bluff` then `tmux send-keys -t bluff "cd ~/Desktop/Play-gent && source .venv/bin/activate && PYTHONPATH=. python training/train_bluff_classifier.py 2>&1 | tee training/bluff_training.log" Enter`
+3. **After bluff finishes, unified training:** `tmux new-session -d -s unified` then `tmux send-keys -t unified "cd ~/Desktop/Play-gent && source .venv/bin/activate && PYTHONPATH=. python training/train_unified.py 2>&1 | tee training/unified_training.log" Enter`
+
+**Monitor tmux:** `tmux attach -t bluff` or `tmux attach -t unified` to watch; detach with Ctrl+B, D. List sessions: `tmux list-sessions`.

training/bluff_training.log

@@ -1,16 +0,0 @@
-
-DistilBertModel LOAD REPORT from: distilbert-base-uncased
-Key                     | Status     |  | 
-------------------------+------------+--+-
-vocab_transform.weight  | UNEXPECTED |  | 
-vocab_projector.bias    | UNEXPECTED |  | 
-vocab_transform.bias    | UNEXPECTED |  | 
-vocab_layer_norm.bias   | UNEXPECTED |  | 
-vocab_layer_norm.weight | UNEXPECTED |  | 
-
-Notes:
-- UNEXPECTED	:can be ignored when loading from different task/architecture; not ok if you expect identical arch.
-Epoch 1/3  Val accuracy: 0.9999  Val F1: 0.9981
-Epoch 2/3  Val accuracy: 1.0000  Val F1: 1.0000
-Epoch 3/3  Val accuracy: 1.0000  Val F1: 1.0000
-Saved model to /home/rayyan/Desktop/Play-gent/training/checkpoints/bluff_classifier.pt, tokenizer to /home/rayyan/Desktop/Play-gent/training/checkpoints/bluff_classifier_tokenizer

training/checkpoints/bluff_classifier_tokenizer/tokenizer_config.json

@@ -1,14 +0,0 @@
-{
-  "backend": "tokenizers",
-  "cls_token": "[CLS]",
-  "do_lower_case": true,
-  "is_local": false,
-  "mask_token": "[MASK]",
-  "model_max_length": 512,
-  "pad_token": "[PAD]",
-  "sep_token": "[SEP]",
-  "strip_accents": null,
-  "tokenize_chinese_chars": true,
-  "tokenizer_class": "BertTokenizer",
-  "unk_token": "[UNK]"
-}

training/checkpoints/phase2_final/README.md

@@ -1,67 +0,0 @@
----
-library_name: transformers
-model_name: phase2_final
-tags:
-- generated_from_trainer
-- trl
-- grpo
-licence: license
----
-
-# Model Card for phase2_final
-
-This model is a fine-tuned version of [None](https://huggingface.co/None).
-It has been trained using [TRL](https://github.com/huggingface/trl).
-
-## Quick start
-
-```python
-from transformers import pipeline
-
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="None", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
-
-## Training procedure
-
- 
-
-
-
-This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
-
-### Framework versions
-
-- TRL: 0.29.0
-- Transformers: 5.3.0
-- Pytorch: 2.12.0.dev20260307+cu128
-- Datasets: 4.6.1
-- Tokenizers: 0.22.2
-
-## Citations
-
-Cite GRPO as:
-
-```bibtex
-@article{shao2024deepseekmath,
-    title        = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
-    author       = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
-    year         = 2024,
-    eprint       = {arXiv:2402.03300},
-}
-
-```
-
-Cite TRL as:
-    
-```bibtex
-@software{vonwerra2020trl,
-  title   = {{TRL: Transformers Reinforcement Learning}},
-  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
-  license = {Apache-2.0},
-  url     = {https://github.com/huggingface/trl},
-  year    = {2020}
-}
-```

training/checkpoints/phase2_final/chat_template.jinja

@@ -1,15 +0,0 @@
-{% for message in messages %}
-{% if message['role'] == 'user' %}
-{{ '<|user|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'system' %}
-{{ '<|system|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'assistant' %}
-{{ '<|assistant|>
-'  + message['content'] + eos_token }}
-{% endif %}
-{% if loop.last and add_generation_prompt %}
-{{ '<|assistant|>' }}
-{% endif %}
-{% endfor %}

training/checkpoints/phase2_final/checkpoint-100/chat_template.jinja

@@ -1,15 +0,0 @@
-{% for message in messages %}
-{% if message['role'] == 'user' %}
-{{ '<|user|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'system' %}
-{{ '<|system|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'assistant' %}
-{{ '<|assistant|>
-'  + message['content'] + eos_token }}
-{% endif %}
-{% if loop.last and add_generation_prompt %}
-{{ '<|assistant|>' }}
-{% endif %}
-{% endfor %}

training/checkpoints/phase2_final/checkpoint-100/config.json

@@ -1,32 +0,0 @@
-{
-  "architectures": [
-    "LlamaForCausalLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "float32",
-  "eos_token_id": 2,
-  "head_dim": 64,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 5632,
-  "max_position_embeddings": 2048,
-  "mlp_bias": false,
-  "model_type": "llama",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 22,
-  "num_key_value_heads": 4,
-  "pad_token_id": 2,
-  "pretraining_tp": 1,
-  "rms_norm_eps": 1e-05,
-  "rope_parameters": {
-    "rope_theta": 10000.0,
-    "rope_type": "default"
-  },
-  "tie_word_embeddings": false,
-  "transformers_version": "5.3.0",
-  "use_cache": false,
-  "vocab_size": 32000
-}

training/checkpoints/phase2_final/checkpoint-100/generation_config.json

@@ -1,9 +0,0 @@
-{
-  "bos_token_id": 1,
-  "eos_token_id": [
-    2
-  ],
-  "max_length": 2048,
-  "pad_token_id": 2,
-  "transformers_version": "5.3.0"
-}

training/checkpoints/phase2_final/checkpoint-100/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "add_prefix_space": null,
-  "backend": "tokenizers",
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "</s>",
-  "is_local": true,
-  "max_length": null,
-  "model_max_length": 2048,
-  "pad_to_multiple_of": null,
-  "pad_token": "</s>",
-  "pad_token_type_id": 0,
-  "padding_side": "left",
-  "sp_model_kwargs": {},
-  "tokenizer_class": "LlamaTokenizer",
-  "truncation_side": "left",
-  "unk_token": "<unk>",
-  "use_default_system_prompt": false
-}

training/checkpoints/phase2_final/checkpoint-100/trainer_state.json

@@ -1,304 +0,0 @@
-{
-  "best_global_step": null,
-  "best_metric": null,
-  "best_model_checkpoint": null,
-  "epoch": 0.1,
-  "eval_steps": 500,
-  "global_step": 100,
-  "is_hyper_param_search": false,
-  "is_local_process_zero": true,
-  "is_world_process_zero": true,
-  "log_history": [
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7924717187881469,
-      "epoch": 0.01,
-      "frac_reward_zero_std": 0.45,
-      "grad_norm": 1.5374246835708618,
-      "learning_rate": 4.775e-06,
-      "loss": 1.4901161193847657e-09,
-      "num_tokens": 35664.0,
-      "reward": 0.11875000391155481,
-      "reward_std": 0.09771842509508133,
-      "rewards/compute_reward/mean": 0.11875000391155481,
-      "rewards/compute_reward/std": 0.09771843403577804,
-      "step": 10,
-      "step_time": 15.109664801302278
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.8351163290441036,
-      "epoch": 0.02,
-      "frac_reward_zero_std": 0.65,
-      "grad_norm": 0.0,
-      "learning_rate": 4.525000000000001e-06,
-      "loss": 2.6822090148925782e-08,
-      "num_tokens": 70060.0,
-      "reward": 0.15750000774860382,
-      "reward_std": 0.04840061739087105,
-      "rewards/compute_reward/mean": 0.15750000774860382,
-      "rewards/compute_reward/std": 0.04840061739087105,
-      "step": 20,
-      "step_time": 14.928892047195404
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.41533662043511865,
-      "epoch": 0.03,
-      "frac_reward_zero_std": 0.8,
-      "grad_norm": 0.0,
-      "learning_rate": 4.2750000000000006e-06,
-      "loss": 1.4901161193847657e-09,
-      "num_tokens": 105588.0,
-      "reward": 0.06375000178813935,
-      "reward_std": 0.04330107718706131,
-      "rewards/compute_reward/mean": 0.06375000178813935,
-      "rewards/compute_reward/std": 0.04330108165740967,
-      "step": 30,
-      "step_time": 15.109792457801813
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.246315559744835,
-      "epoch": 0.04,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 4.0250000000000004e-06,
-      "loss": 0.0,
-      "num_tokens": 141264.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 40,
-      "step_time": 15.195196880902222
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7081560462713241,
-      "epoch": 0.05,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.7750000000000003e-06,
-      "loss": 0.0,
-      "num_tokens": 176780.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 50,
-      "step_time": 15.140776808797819
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.727844113111496,
-      "epoch": 0.06,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.525e-06,
-      "loss": 0.0,
-      "num_tokens": 212628.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 60,
-      "step_time": 15.286061269601486
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7312307402491569,
-      "epoch": 0.07,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.2750000000000004e-06,
-      "loss": 0.0,
-      "num_tokens": 248212.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 70,
-      "step_time": 15.278303197700733
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7322262570261955,
-      "epoch": 0.08,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.0250000000000003e-06,
-      "loss": 0.0,
-      "num_tokens": 283644.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 80,
-      "step_time": 15.146252356799959
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7361132100224494,
-      "epoch": 0.09,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 2.7750000000000005e-06,
-      "loss": 0.0,
-      "num_tokens": 318532.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 90,
-      "step_time": 15.026733554197563
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7636664807796478,
-      "epoch": 0.1,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 2.5250000000000004e-06,
-      "loss": 0.0,
-      "num_tokens": 355352.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 100,
-      "step_time": 15.381215008600702
-    }
-  ],
-  "logging_steps": 10,
-  "max_steps": 200,
-  "num_input_tokens_seen": 355352,
-  "num_train_epochs": 1,
-  "save_steps": 100,
-  "stateful_callbacks": {
-    "TrainerControl": {
-      "args": {
-        "should_epoch_stop": false,
-        "should_evaluate": false,
-        "should_log": false,
-        "should_save": true,
-        "should_training_stop": false
-      },
-      "attributes": {}
-    }
-  },
-  "total_flos": 0.0,
-  "train_batch_size": 2,
-  "trial_name": null,
-  "trial_params": null
-}

training/checkpoints/phase2_final/checkpoint-200/chat_template.jinja

@@ -1,15 +0,0 @@
-{% for message in messages %}
-{% if message['role'] == 'user' %}
-{{ '<|user|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'system' %}
-{{ '<|system|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'assistant' %}
-{{ '<|assistant|>
-'  + message['content'] + eos_token }}
-{% endif %}
-{% if loop.last and add_generation_prompt %}
-{{ '<|assistant|>' }}
-{% endif %}
-{% endfor %}

training/checkpoints/phase2_final/checkpoint-200/config.json

@@ -1,32 +0,0 @@
-{
-  "architectures": [
-    "LlamaForCausalLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "float32",
-  "eos_token_id": 2,
-  "head_dim": 64,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 5632,
-  "max_position_embeddings": 2048,
-  "mlp_bias": false,
-  "model_type": "llama",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 22,
-  "num_key_value_heads": 4,
-  "pad_token_id": 2,
-  "pretraining_tp": 1,
-  "rms_norm_eps": 1e-05,
-  "rope_parameters": {
-    "rope_theta": 10000.0,
-    "rope_type": "default"
-  },
-  "tie_word_embeddings": false,
-  "transformers_version": "5.3.0",
-  "use_cache": false,
-  "vocab_size": 32000
-}

training/checkpoints/phase2_final/checkpoint-200/generation_config.json

@@ -1,9 +0,0 @@
-{
-  "bos_token_id": 1,
-  "eos_token_id": [
-    2
-  ],
-  "max_length": 2048,
-  "pad_token_id": 2,
-  "transformers_version": "5.3.0"
-}

training/checkpoints/phase2_final/checkpoint-200/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "add_prefix_space": null,
-  "backend": "tokenizers",
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "</s>",
-  "is_local": true,
-  "max_length": null,
-  "model_max_length": 2048,
-  "pad_to_multiple_of": null,
-  "pad_token": "</s>",
-  "pad_token_type_id": 0,
-  "padding_side": "left",
-  "sp_model_kwargs": {},
-  "tokenizer_class": "LlamaTokenizer",
-  "truncation_side": "left",
-  "unk_token": "<unk>",
-  "use_default_system_prompt": false
-}

training/checkpoints/phase2_final/checkpoint-200/trainer_state.json

@@ -1,574 +0,0 @@
-{
-  "best_global_step": null,
-  "best_metric": null,
-  "best_model_checkpoint": null,
-  "epoch": 0.2,
-  "eval_steps": 500,
-  "global_step": 200,
-  "is_hyper_param_search": false,
-  "is_local_process_zero": true,
-  "is_world_process_zero": true,
-  "log_history": [
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7924717187881469,
-      "epoch": 0.01,
-      "frac_reward_zero_std": 0.45,
-      "grad_norm": 1.5374246835708618,
-      "learning_rate": 4.775e-06,
-      "loss": 1.4901161193847657e-09,
-      "num_tokens": 35664.0,
-      "reward": 0.11875000391155481,
-      "reward_std": 0.09771842509508133,
-      "rewards/compute_reward/mean": 0.11875000391155481,
-      "rewards/compute_reward/std": 0.09771843403577804,
-      "step": 10,
-      "step_time": 15.109664801302278
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.8351163290441036,
-      "epoch": 0.02,
-      "frac_reward_zero_std": 0.65,
-      "grad_norm": 0.0,
-      "learning_rate": 4.525000000000001e-06,
-      "loss": 2.6822090148925782e-08,
-      "num_tokens": 70060.0,
-      "reward": 0.15750000774860382,
-      "reward_std": 0.04840061739087105,
-      "rewards/compute_reward/mean": 0.15750000774860382,
-      "rewards/compute_reward/std": 0.04840061739087105,
-      "step": 20,
-      "step_time": 14.928892047195404
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.41533662043511865,
-      "epoch": 0.03,
-      "frac_reward_zero_std": 0.8,
-      "grad_norm": 0.0,
-      "learning_rate": 4.2750000000000006e-06,
-      "loss": 1.4901161193847657e-09,
-      "num_tokens": 105588.0,
-      "reward": 0.06375000178813935,
-      "reward_std": 0.04330107718706131,
-      "rewards/compute_reward/mean": 0.06375000178813935,
-      "rewards/compute_reward/std": 0.04330108165740967,
-      "step": 30,
-      "step_time": 15.109792457801813
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.246315559744835,
-      "epoch": 0.04,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 4.0250000000000004e-06,
-      "loss": 0.0,
-      "num_tokens": 141264.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 40,
-      "step_time": 15.195196880902222
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7081560462713241,
-      "epoch": 0.05,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.7750000000000003e-06,
-      "loss": 0.0,
-      "num_tokens": 176780.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 50,
-      "step_time": 15.140776808797819
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.727844113111496,
-      "epoch": 0.06,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.525e-06,
-      "loss": 0.0,
-      "num_tokens": 212628.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 60,
-      "step_time": 15.286061269601486
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7312307402491569,
-      "epoch": 0.07,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.2750000000000004e-06,
-      "loss": 0.0,
-      "num_tokens": 248212.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 70,
-      "step_time": 15.278303197700733
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7322262570261955,
-      "epoch": 0.08,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 3.0250000000000003e-06,
-      "loss": 0.0,
-      "num_tokens": 283644.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 80,
-      "step_time": 15.146252356799959
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7361132100224494,
-      "epoch": 0.09,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 2.7750000000000005e-06,
-      "loss": 0.0,
-      "num_tokens": 318532.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 90,
-      "step_time": 15.026733554197563
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7636664807796478,
-      "epoch": 0.1,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 2.5250000000000004e-06,
-      "loss": 0.0,
-      "num_tokens": 355352.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 100,
-      "step_time": 15.381215008600702
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7429351836442948,
-      "epoch": 0.11,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 2.2750000000000002e-06,
-      "loss": 0.0,
-      "num_tokens": 389508.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 110,
-      "step_time": 15.039604106301704
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7703481003642082,
-      "epoch": 0.12,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 2.025e-06,
-      "loss": 0.0,
-      "num_tokens": 426240.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 120,
-      "step_time": 15.29271342299835
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.7375139251351357,
-      "epoch": 0.13,
-      "frac_reward_zero_std": 1.0,
-      "grad_norm": 0.0,
-      "learning_rate": 1.7750000000000002e-06,
-      "loss": 0.0,
-      "num_tokens": 462400.0,
-      "reward": 0.30000001192092896,
-      "reward_std": 0.0,
-      "rewards/compute_reward/mean": 0.30000001192092896,
-      "rewards/compute_reward/std": 0.0,
-      "step": 130,
-      "step_time": 15.20639470120077
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.8568216070532799,
-      "epoch": 0.14,
-      "frac_reward_zero_std": 0.9,
-      "grad_norm": 0.0,
-      "learning_rate": 1.525e-06,
-      "loss": 1.7881393432617187e-08,
-      "num_tokens": 498020.0,
-      "reward": 0.30500001311302183,
-      "reward_std": 0.01414213478565216,
-      "rewards/compute_reward/mean": 0.30500001311302183,
-      "rewards/compute_reward/std": 0.01414213478565216,
-      "step": 140,
-      "step_time": 15.200056954801402
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.4760520339012146,
-      "epoch": 0.15,
-      "frac_reward_zero_std": 0.95,
-      "grad_norm": 0.0,
-      "learning_rate": 1.275e-06,
-      "loss": 8.940696716308593e-09,
-      "num_tokens": 532668.0,
-      "reward": 0.3025000125169754,
-      "reward_std": 0.00707106739282608,
-      "rewards/compute_reward/mean": 0.3025000125169754,
-      "rewards/compute_reward/std": 0.00707106739282608,
-      "step": 150,
-      "step_time": 14.748404727898015
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.7379814833402634,
-      "epoch": 0.16,
-      "frac_reward_zero_std": 0.9,
-      "grad_norm": 0.0,
-      "learning_rate": 1.025e-06,
-      "loss": 1.564621925354004e-08,
-      "num_tokens": 567544.0,
-      "reward": 0.3037500113248825,
-      "reward_std": 0.01060660146176815,
-      "rewards/compute_reward/mean": 0.3037500113248825,
-      "rewards/compute_reward/std": 0.01060660108923912,
-      "step": 160,
-      "step_time": 15.037257523898734
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.5534777998924256,
-      "epoch": 0.17,
-      "frac_reward_zero_std": 0.8,
-      "grad_norm": 0.0,
-      "learning_rate": 7.750000000000001e-07,
-      "loss": 1.7881393432617187e-08,
-      "num_tokens": 604400.0,
-      "reward": 0.31500001549720763,
-      "reward_std": 0.032658536732196805,
-      "rewards/compute_reward/mean": 0.31500001549720763,
-      "rewards/compute_reward/std": 0.032658536732196805,
-      "step": 170,
-      "step_time": 15.339705387198773
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.3570319384336471,
-      "epoch": 0.18,
-      "frac_reward_zero_std": 0.9,
-      "grad_norm": 2.3024227619171143,
-      "learning_rate": 5.250000000000001e-07,
-      "loss": 8.195638656616212e-09,
-      "num_tokens": 639432.0,
-      "reward": 0.3075000137090683,
-      "reward_std": 0.02121320217847824,
-      "rewards/compute_reward/mean": 0.3075000137090683,
-      "rewards/compute_reward/std": 0.02121320217847824,
-      "step": 180,
-      "step_time": 14.838772397398861
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.4456530869007111,
-      "epoch": 0.19,
-      "frac_reward_zero_std": 0.75,
-      "grad_norm": 1.3511810302734375,
-      "learning_rate": 2.75e-07,
-      "loss": 4.0978193283081055e-08,
-      "num_tokens": 674972.0,
-      "reward": 0.31125001311302186,
-      "reward_std": 0.03181980364024639,
-      "rewards/compute_reward/mean": 0.31125001311302186,
-      "rewards/compute_reward/std": 0.03181980326771736,
-      "step": 190,
-      "step_time": 15.081224197598932
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.5674545228481294,
-      "epoch": 0.2,
-      "frac_reward_zero_std": 0.75,
-      "grad_norm": 1.818772792816162,
-      "learning_rate": 2.5000000000000002e-08,
-      "loss": 3.129243850708008e-08,
-      "num_tokens": 709480.0,
-      "reward": 0.31750001311302184,
-      "reward_std": 0.04316474497318268,
-      "rewards/compute_reward/mean": 0.31750001311302184,
-      "rewards/compute_reward/std": 0.04316474497318268,
-      "step": 200,
-      "step_time": 15.07085579989798
-    }
-  ],
-  "logging_steps": 10,
-  "max_steps": 200,
-  "num_input_tokens_seen": 709480,
-  "num_train_epochs": 1,
-  "save_steps": 100,
-  "stateful_callbacks": {
-    "TrainerControl": {
-      "args": {
-        "should_epoch_stop": false,
-        "should_evaluate": false,
-        "should_log": false,
-        "should_save": true,
-        "should_training_stop": true
-      },
-      "attributes": {}
-    }
-  },
-  "total_flos": 0.0,
-  "train_batch_size": 2,
-  "trial_name": null,
-  "trial_params": null
-}

training/checkpoints/phase2_final/config.json

@@ -1,32 +0,0 @@
-{
-  "architectures": [
-    "LlamaForCausalLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "float32",
-  "eos_token_id": 2,
-  "head_dim": 64,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 5632,
-  "max_position_embeddings": 2048,
-  "mlp_bias": false,
-  "model_type": "llama",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 22,
-  "num_key_value_heads": 4,
-  "pad_token_id": 2,
-  "pretraining_tp": 1,
-  "rms_norm_eps": 1e-05,
-  "rope_parameters": {
-    "rope_theta": 10000.0,
-    "rope_type": "default"
-  },
-  "tie_word_embeddings": false,
-  "transformers_version": "5.3.0",
-  "use_cache": false,
-  "vocab_size": 32000
-}

training/checkpoints/phase2_final/generation_config.json

@@ -1,9 +0,0 @@
-{
-  "bos_token_id": 1,
-  "eos_token_id": [
-    2
-  ],
-  "max_length": 2048,
-  "pad_token_id": 2,
-  "transformers_version": "5.3.0"
-}

training/checkpoints/phase2_final/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "add_prefix_space": null,
-  "backend": "tokenizers",
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "</s>",
-  "is_local": true,
-  "max_length": null,
-  "model_max_length": 2048,
-  "pad_to_multiple_of": null,
-  "pad_token": "</s>",
-  "pad_token_type_id": 0,
-  "padding_side": "left",
-  "sp_model_kwargs": {},
-  "tokenizer_class": "LlamaTokenizer",
-  "truncation_side": "left",
-  "unk_token": "<unk>",
-  "use_default_system_prompt": false
-}

training/checkpoints/unified_final/README.md

@@ -1,67 +0,0 @@
----
-library_name: transformers
-model_name: unified_final
-tags:
-- generated_from_trainer
-- trl
-- grpo
-licence: license
----
-
-# Model Card for unified_final
-
-This model is a fine-tuned version of [None](https://huggingface.co/None).
-It has been trained using [TRL](https://github.com/huggingface/trl).
-
-## Quick start
-
-```python
-from transformers import pipeline
-
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="None", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
-
-## Training procedure
-
- 
-
-
-
-This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
-
-### Framework versions
-
-- TRL: 0.29.0
-- Transformers: 5.3.0
-- Pytorch: 2.12.0.dev20260307+cu128
-- Datasets: 4.6.1
-- Tokenizers: 0.22.2
-
-## Citations
-
-Cite GRPO as:
-
-```bibtex
-@article{shao2024deepseekmath,
-    title        = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
-    author       = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
-    year         = 2024,
-    eprint       = {arXiv:2402.03300},
-}
-
-```
-
-Cite TRL as:
-    
-```bibtex
-@software{vonwerra2020trl,
-  title   = {{TRL: Transformers Reinforcement Learning}},
-  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
-  license = {Apache-2.0},
-  url     = {https://github.com/huggingface/trl},
-  year    = {2020}
-}
-```

training/checkpoints/unified_final/chat_template.jinja

@@ -1,15 +0,0 @@
-{% for message in messages %}
-{% if message['role'] == 'user' %}
-{{ '<|user|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'system' %}
-{{ '<|system|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'assistant' %}
-{{ '<|assistant|>
-'  + message['content'] + eos_token }}
-{% endif %}
-{% if loop.last and add_generation_prompt %}
-{{ '<|assistant|>' }}
-{% endif %}
-{% endfor %}

training/checkpoints/unified_final/checkpoint-100/chat_template.jinja

@@ -1,15 +0,0 @@
-{% for message in messages %}
-{% if message['role'] == 'user' %}
-{{ '<|user|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'system' %}
-{{ '<|system|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'assistant' %}
-{{ '<|assistant|>
-'  + message['content'] + eos_token }}
-{% endif %}
-{% if loop.last and add_generation_prompt %}
-{{ '<|assistant|>' }}
-{% endif %}
-{% endfor %}

training/checkpoints/unified_final/checkpoint-100/config.json

@@ -1,32 +0,0 @@
-{
-  "architectures": [
-    "LlamaForCausalLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "float32",
-  "eos_token_id": 2,
-  "head_dim": 64,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 5632,
-  "max_position_embeddings": 2048,
-  "mlp_bias": false,
-  "model_type": "llama",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 22,
-  "num_key_value_heads": 4,
-  "pad_token_id": 2,
-  "pretraining_tp": 1,
-  "rms_norm_eps": 1e-05,
-  "rope_parameters": {
-    "rope_theta": 10000.0,
-    "rope_type": "default"
-  },
-  "tie_word_embeddings": false,
-  "transformers_version": "5.3.0",
-  "use_cache": false,
-  "vocab_size": 32000
-}

training/checkpoints/unified_final/checkpoint-100/generation_config.json

@@ -1,9 +0,0 @@
-{
-  "bos_token_id": 1,
-  "eos_token_id": [
-    2
-  ],
-  "max_length": 2048,
-  "pad_token_id": 2,
-  "transformers_version": "5.3.0"
-}

training/checkpoints/unified_final/checkpoint-100/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "add_prefix_space": null,
-  "backend": "tokenizers",
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "</s>",
-  "is_local": true,
-  "max_length": null,
-  "model_max_length": 2048,
-  "pad_to_multiple_of": null,
-  "pad_token": "</s>",
-  "pad_token_type_id": 0,
-  "padding_side": "left",
-  "sp_model_kwargs": {},
-  "tokenizer_class": "LlamaTokenizer",
-  "truncation_side": "left",
-  "unk_token": "<unk>",
-  "use_default_system_prompt": false
-}

training/checkpoints/unified_final/checkpoint-100/trainer_state.json

@@ -1,304 +0,0 @@
-{
-  "best_global_step": null,
-  "best_metric": null,
-  "best_model_checkpoint": null,
-  "epoch": 0.1,
-  "eval_steps": 500,
-  "global_step": 100,
-  "is_hyper_param_search": false,
-  "is_local_process_zero": true,
-  "is_world_process_zero": true,
-  "log_history": [
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.3566992908716202,
-      "epoch": 0.01,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 0.7344621419906616,
-      "learning_rate": 4.775e-06,
-      "loss": 3.0994415283203126e-07,
-      "num_tokens": 35800.0,
-      "reward": 0.01268580500036478,
-      "reward_std": 0.02462496655061841,
-      "rewards/compute_reward/mean": 0.01268580500036478,
-      "rewards/compute_reward/std": 0.024624967435374855,
-      "step": 10,
-      "step_time": 10.7134718033034
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.3169427752494811,
-      "epoch": 0.02,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 4.441726207733154,
-      "learning_rate": 4.525000000000001e-06,
-      "loss": -4.246830940246582e-07,
-      "num_tokens": 71748.0,
-      "reward": -0.04455982223153114,
-      "reward_std": 0.035665383422747256,
-      "rewards/compute_reward/mean": -0.04455982223153114,
-      "rewards/compute_reward/std": 0.03566538490122184,
-      "step": 20,
-      "step_time": 10.643421414200565
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 8.9,
-      "completions/mean_length": 99.8625,
-      "completions/mean_terminated_length": 8.9,
-      "completions/min_length": 98.9,
-      "completions/min_terminated_length": 8.9,
-      "entropy": 1.0057833462953567,
-      "epoch": 0.03,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.0170326232910156,
-      "learning_rate": 4.2750000000000006e-06,
-      "loss": -0.0018164031207561493,
-      "num_tokens": 108181.0,
-      "reward": 0.0374881561845541,
-      "reward_std": 0.020618790527805686,
-      "rewards/compute_reward/mean": 0.0374881561845541,
-      "rewards/compute_reward/std": 0.0206187907140702,
-      "step": 30,
-      "step_time": 10.756140169796709
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 6.6,
-      "completions/mean_length": 99.575,
-      "completions/mean_terminated_length": 6.6,
-      "completions/min_length": 96.6,
-      "completions/min_terminated_length": 6.6,
-      "entropy": 1.7816664546728134,
-      "epoch": 0.04,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 5.86561393737793,
-      "learning_rate": 4.0250000000000004e-06,
-      "loss": -0.006361240148544311,
-      "num_tokens": 143375.0,
-      "reward": -0.014824284799396991,
-      "reward_std": 0.06699581742286682,
-      "rewards/compute_reward/mean": -0.014824284799396991,
-      "rewards/compute_reward/std": 0.06699582003057003,
-      "step": 40,
-      "step_time": 10.785410385398427
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 3.0,
-      "completions/mean_length": 99.125,
-      "completions/mean_terminated_length": 3.0,
-      "completions/min_length": 93.0,
-      "completions/min_terminated_length": 3.0,
-      "entropy": 2.1307705104351045,
-      "epoch": 0.05,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 6.191352367401123,
-      "learning_rate": 3.7750000000000003e-06,
-      "loss": -0.011027154326438905,
-      "num_tokens": 178941.0,
-      "reward": -0.016337488451972602,
-      "reward_std": 0.051818730868399145,
-      "rewards/compute_reward/mean": -0.016337488451972602,
-      "rewards/compute_reward/std": 0.05181873142719269,
-      "step": 50,
-      "step_time": 10.741381045605522
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 8.8,
-      "completions/mean_length": 99.85,
-      "completions/mean_terminated_length": 8.8,
-      "completions/min_length": 98.8,
-      "completions/min_terminated_length": 8.8,
-      "entropy": 2.1041357040405275,
-      "epoch": 0.06,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 8.536041259765625,
-      "learning_rate": 3.525e-06,
-      "loss": 0.0019509844481945039,
-      "num_tokens": 216257.0,
-      "reward": 0.035917540453374384,
-      "reward_std": 0.04930563308298588,
-      "rewards/compute_reward/mean": 0.035917540453374384,
-      "rewards/compute_reward/std": 0.049305635318160054,
-      "step": 60,
-      "step_time": 11.27133785020269
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.8,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 48.2,
-      "completions/mean_length": 92.9625,
-      "completions/mean_terminated_length": 38.51333351135254,
-      "completions/min_length": 70.1,
-      "completions/min_terminated_length": 30.1,
-      "entropy": 1.6469052851200103,
-      "epoch": 0.07,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 6.919373512268066,
-      "learning_rate": 3.2750000000000004e-06,
-      "loss": -0.02075239419937134,
-      "num_tokens": 251110.0,
-      "reward": 0.007261525164358318,
-      "reward_std": 0.0802696269005537,
-      "rewards/compute_reward/mean": 0.007261525164358318,
-      "rewards/compute_reward/std": 0.08026962876319885,
-      "step": 70,
-      "step_time": 10.774873650902009
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 3.1,
-      "completions/mean_length": 99.1375,
-      "completions/mean_terminated_length": 3.1,
-      "completions/min_length": 93.1,
-      "completions/min_terminated_length": 3.1,
-      "entropy": 2.2336367428302766,
-      "epoch": 0.08,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 4.918172836303711,
-      "learning_rate": 3.0250000000000003e-06,
-      "loss": 0.008250368386507034,
-      "num_tokens": 285729.0,
-      "reward": 0.027657157555222512,
-      "reward_std": 0.04840414375066757,
-      "rewards/compute_reward/mean": 0.027657157555222512,
-      "rewards/compute_reward/std": 0.048404145427048205,
-      "step": 80,
-      "step_time": 10.43483721170196
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.8057245463132858,
-      "epoch": 0.09,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 4.417481422424316,
-      "learning_rate": 2.7750000000000005e-06,
-      "loss": 2.216547727584839e-08,
-      "num_tokens": 320249.0,
-      "reward": 0.07908838111907243,
-      "reward_std": 0.07920666746795177,
-      "rewards/compute_reward/mean": 0.07908838111907243,
-      "rewards/compute_reward/std": 0.07920666970312595,
-      "step": 90,
-      "step_time": 10.337220244196942
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.4064194440841675,
-      "epoch": 0.1,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.352966785430908,
-      "learning_rate": 2.5250000000000004e-06,
-      "loss": 8.493661880493164e-08,
-      "num_tokens": 355369.0,
-      "reward": 0.14763977155089378,
-      "reward_std": 0.07424246501177549,
-      "rewards/compute_reward/mean": 0.14763977155089378,
-      "rewards/compute_reward/std": 0.0742424676194787,
-      "step": 100,
-      "step_time": 10.74917738300719
-    }
-  ],
-  "logging_steps": 10,
-  "max_steps": 200,
-  "num_input_tokens_seen": 355369,
-  "num_train_epochs": 1,
-  "save_steps": 100,
-  "stateful_callbacks": {
-    "TrainerControl": {
-      "args": {
-        "should_epoch_stop": false,
-        "should_evaluate": false,
-        "should_log": false,
-        "should_save": true,
-        "should_training_stop": false
-      },
-      "attributes": {}
-    }
-  },
-  "total_flos": 0.0,
-  "train_batch_size": 2,
-  "trial_name": null,
-  "trial_params": null
-}

training/checkpoints/unified_final/checkpoint-200/chat_template.jinja

@@ -1,15 +0,0 @@
-{% for message in messages %}
-{% if message['role'] == 'user' %}
-{{ '<|user|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'system' %}
-{{ '<|system|>
-' + message['content'] + eos_token }}
-{% elif message['role'] == 'assistant' %}
-{{ '<|assistant|>
-'  + message['content'] + eos_token }}
-{% endif %}
-{% if loop.last and add_generation_prompt %}
-{{ '<|assistant|>' }}
-{% endif %}
-{% endfor %}

training/checkpoints/unified_final/checkpoint-200/config.json

@@ -1,32 +0,0 @@
-{
-  "architectures": [
-    "LlamaForCausalLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "float32",
-  "eos_token_id": 2,
-  "head_dim": 64,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 5632,
-  "max_position_embeddings": 2048,
-  "mlp_bias": false,
-  "model_type": "llama",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 22,
-  "num_key_value_heads": 4,
-  "pad_token_id": 2,
-  "pretraining_tp": 1,
-  "rms_norm_eps": 1e-05,
-  "rope_parameters": {
-    "rope_theta": 10000.0,
-    "rope_type": "default"
-  },
-  "tie_word_embeddings": false,
-  "transformers_version": "5.3.0",
-  "use_cache": false,
-  "vocab_size": 32000
-}

training/checkpoints/unified_final/checkpoint-200/generation_config.json

@@ -1,9 +0,0 @@
-{
-  "bos_token_id": 1,
-  "eos_token_id": [
-    2
-  ],
-  "max_length": 2048,
-  "pad_token_id": 2,
-  "transformers_version": "5.3.0"
-}

training/checkpoints/unified_final/checkpoint-200/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "add_prefix_space": null,
-  "backend": "tokenizers",
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "</s>",
-  "is_local": true,
-  "max_length": null,
-  "model_max_length": 2048,
-  "pad_to_multiple_of": null,
-  "pad_token": "</s>",
-  "pad_token_type_id": 0,
-  "padding_side": "left",
-  "sp_model_kwargs": {},
-  "tokenizer_class": "LlamaTokenizer",
-  "truncation_side": "left",
-  "unk_token": "<unk>",
-  "use_default_system_prompt": false
-}

training/checkpoints/unified_final/checkpoint-200/trainer_state.json

@@ -1,574 +0,0 @@
-{
-  "best_global_step": null,
-  "best_metric": null,
-  "best_model_checkpoint": null,
-  "epoch": 0.2,
-  "eval_steps": 500,
-  "global_step": 200,
-  "is_hyper_param_search": false,
-  "is_local_process_zero": true,
-  "is_world_process_zero": true,
-  "log_history": [
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.3566992908716202,
-      "epoch": 0.01,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 0.7344621419906616,
-      "learning_rate": 4.775e-06,
-      "loss": 3.0994415283203126e-07,
-      "num_tokens": 35800.0,
-      "reward": 0.01268580500036478,
-      "reward_std": 0.02462496655061841,
-      "rewards/compute_reward/mean": 0.01268580500036478,
-      "rewards/compute_reward/std": 0.024624967435374855,
-      "step": 10,
-      "step_time": 10.7134718033034
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.3169427752494811,
-      "epoch": 0.02,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 4.441726207733154,
-      "learning_rate": 4.525000000000001e-06,
-      "loss": -4.246830940246582e-07,
-      "num_tokens": 71748.0,
-      "reward": -0.04455982223153114,
-      "reward_std": 0.035665383422747256,
-      "rewards/compute_reward/mean": -0.04455982223153114,
-      "rewards/compute_reward/std": 0.03566538490122184,
-      "step": 20,
-      "step_time": 10.643421414200565
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 8.9,
-      "completions/mean_length": 99.8625,
-      "completions/mean_terminated_length": 8.9,
-      "completions/min_length": 98.9,
-      "completions/min_terminated_length": 8.9,
-      "entropy": 1.0057833462953567,
-      "epoch": 0.03,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.0170326232910156,
-      "learning_rate": 4.2750000000000006e-06,
-      "loss": -0.0018164031207561493,
-      "num_tokens": 108181.0,
-      "reward": 0.0374881561845541,
-      "reward_std": 0.020618790527805686,
-      "rewards/compute_reward/mean": 0.0374881561845541,
-      "rewards/compute_reward/std": 0.0206187907140702,
-      "step": 30,
-      "step_time": 10.756140169796709
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 6.6,
-      "completions/mean_length": 99.575,
-      "completions/mean_terminated_length": 6.6,
-      "completions/min_length": 96.6,
-      "completions/min_terminated_length": 6.6,
-      "entropy": 1.7816664546728134,
-      "epoch": 0.04,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 5.86561393737793,
-      "learning_rate": 4.0250000000000004e-06,
-      "loss": -0.006361240148544311,
-      "num_tokens": 143375.0,
-      "reward": -0.014824284799396991,
-      "reward_std": 0.06699581742286682,
-      "rewards/compute_reward/mean": -0.014824284799396991,
-      "rewards/compute_reward/std": 0.06699582003057003,
-      "step": 40,
-      "step_time": 10.785410385398427
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 3.0,
-      "completions/mean_length": 99.125,
-      "completions/mean_terminated_length": 3.0,
-      "completions/min_length": 93.0,
-      "completions/min_terminated_length": 3.0,
-      "entropy": 2.1307705104351045,
-      "epoch": 0.05,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 6.191352367401123,
-      "learning_rate": 3.7750000000000003e-06,
-      "loss": -0.011027154326438905,
-      "num_tokens": 178941.0,
-      "reward": -0.016337488451972602,
-      "reward_std": 0.051818730868399145,
-      "rewards/compute_reward/mean": -0.016337488451972602,
-      "rewards/compute_reward/std": 0.05181873142719269,
-      "step": 50,
-      "step_time": 10.741381045605522
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 8.8,
-      "completions/mean_length": 99.85,
-      "completions/mean_terminated_length": 8.8,
-      "completions/min_length": 98.8,
-      "completions/min_terminated_length": 8.8,
-      "entropy": 2.1041357040405275,
-      "epoch": 0.06,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 8.536041259765625,
-      "learning_rate": 3.525e-06,
-      "loss": 0.0019509844481945039,
-      "num_tokens": 216257.0,
-      "reward": 0.035917540453374384,
-      "reward_std": 0.04930563308298588,
-      "rewards/compute_reward/mean": 0.035917540453374384,
-      "rewards/compute_reward/std": 0.049305635318160054,
-      "step": 60,
-      "step_time": 11.27133785020269
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.8,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 48.2,
-      "completions/mean_length": 92.9625,
-      "completions/mean_terminated_length": 38.51333351135254,
-      "completions/min_length": 70.1,
-      "completions/min_terminated_length": 30.1,
-      "entropy": 1.6469052851200103,
-      "epoch": 0.07,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 6.919373512268066,
-      "learning_rate": 3.2750000000000004e-06,
-      "loss": -0.02075239419937134,
-      "num_tokens": 251110.0,
-      "reward": 0.007261525164358318,
-      "reward_std": 0.0802696269005537,
-      "rewards/compute_reward/mean": 0.007261525164358318,
-      "rewards/compute_reward/std": 0.08026962876319885,
-      "step": 70,
-      "step_time": 10.774873650902009
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 3.1,
-      "completions/mean_length": 99.1375,
-      "completions/mean_terminated_length": 3.1,
-      "completions/min_length": 93.1,
-      "completions/min_terminated_length": 3.1,
-      "entropy": 2.2336367428302766,
-      "epoch": 0.08,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 4.918172836303711,
-      "learning_rate": 3.0250000000000003e-06,
-      "loss": 0.008250368386507034,
-      "num_tokens": 285729.0,
-      "reward": 0.027657157555222512,
-      "reward_std": 0.04840414375066757,
-      "rewards/compute_reward/mean": 0.027657157555222512,
-      "rewards/compute_reward/std": 0.048404145427048205,
-      "step": 80,
-      "step_time": 10.43483721170196
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.8057245463132858,
-      "epoch": 0.09,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 4.417481422424316,
-      "learning_rate": 2.7750000000000005e-06,
-      "loss": 2.216547727584839e-08,
-      "num_tokens": 320249.0,
-      "reward": 0.07908838111907243,
-      "reward_std": 0.07920666746795177,
-      "rewards/compute_reward/mean": 0.07908838111907243,
-      "rewards/compute_reward/std": 0.07920666970312595,
-      "step": 90,
-      "step_time": 10.337220244196942
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.4064194440841675,
-      "epoch": 0.1,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.352966785430908,
-      "learning_rate": 2.5250000000000004e-06,
-      "loss": 8.493661880493164e-08,
-      "num_tokens": 355369.0,
-      "reward": 0.14763977155089378,
-      "reward_std": 0.07424246501177549,
-      "rewards/compute_reward/mean": 0.14763977155089378,
-      "rewards/compute_reward/std": 0.0742424676194787,
-      "step": 100,
-      "step_time": 10.74917738300719
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.2582464694976807,
-      "epoch": 0.11,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.9595463275909424,
-      "learning_rate": 2.2750000000000002e-06,
-      "loss": -3.874301910400391e-08,
-      "num_tokens": 392289.0,
-      "reward": 0.18278183937072753,
-      "reward_std": 0.052620683796703815,
-      "rewards/compute_reward/mean": 0.18278183937072753,
-      "rewards/compute_reward/std": 0.05262068491429091,
-      "step": 110,
-      "step_time": 11.17140419179923
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.8805452413856983,
-      "epoch": 0.12,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 2.707214593887329,
-      "learning_rate": 2.025e-06,
-      "loss": 1.5050172805786132e-07,
-      "num_tokens": 430501.0,
-      "reward": 0.22903144657611846,
-      "reward_std": 0.04029850559309125,
-      "rewards/compute_reward/mean": 0.22903144657611846,
-      "rewards/compute_reward/std": 0.04029850568622351,
-      "step": 120,
-      "step_time": 11.244449263699062
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.8755271568894386,
-      "epoch": 0.13,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.942605495452881,
-      "learning_rate": 1.7750000000000002e-06,
-      "loss": 1.2218952178955077e-07,
-      "num_tokens": 467245.0,
-      "reward": 0.18334048390388488,
-      "reward_std": 0.07254596166312695,
-      "rewards/compute_reward/mean": 0.18334048390388488,
-      "rewards/compute_reward/std": 0.072545962408185,
-      "step": 130,
-      "step_time": 11.071729802998016
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.9737002968788147,
-      "epoch": 0.14,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 4.040837287902832,
-      "learning_rate": 1.525e-06,
-      "loss": -1.4007091522216797e-07,
-      "num_tokens": 503017.0,
-      "reward": 0.20783505886793135,
-      "reward_std": 0.06580547224730253,
-      "rewards/compute_reward/mean": 0.20783505886793135,
-      "rewards/compute_reward/std": 0.06580547466874123,
-      "step": 140,
-      "step_time": 10.841636341501726
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.9901166066527367,
-      "epoch": 0.15,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.720881462097168,
-      "learning_rate": 1.275e-06,
-      "loss": 2.0861625671386717e-08,
-      "num_tokens": 539801.0,
-      "reward": 0.2224348157644272,
-      "reward_std": 0.05879365894943476,
-      "rewards/compute_reward/mean": 0.2224348157644272,
-      "rewards/compute_reward/std": 0.05879366043955088,
-      "step": 150,
-      "step_time": 10.85469058619783
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 1.1208710052073,
-      "epoch": 0.16,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.452557325363159,
-      "learning_rate": 1.025e-06,
-      "loss": 1.4603137969970704e-07,
-      "num_tokens": 575385.0,
-      "reward": 0.1992661789059639,
-      "reward_std": 0.06030977526679635,
-      "rewards/compute_reward/mean": 0.1992661789059639,
-      "rewards/compute_reward/std": 0.060309774987399575,
-      "step": 160,
-      "step_time": 10.620040459206212
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 0.9875,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 8.5,
-      "completions/mean_length": 99.8125,
-      "completions/mean_terminated_length": 8.5,
-      "completions/min_length": 98.5,
-      "completions/min_terminated_length": 8.5,
-      "entropy": 0.943237779289484,
-      "epoch": 0.17,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.998199701309204,
-      "learning_rate": 7.750000000000001e-07,
-      "loss": 0.0005225777626037597,
-      "num_tokens": 611998.0,
-      "reward": 0.21552147567272187,
-      "reward_std": 0.032230423856526615,
-      "rewards/compute_reward/mean": 0.21552147567272187,
-      "rewards/compute_reward/std": 0.0322304243221879,
-      "step": 170,
-      "step_time": 10.901679297701047
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.9798725090920926,
-      "epoch": 0.18,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.732668161392212,
-      "learning_rate": 5.250000000000001e-07,
-      "loss": -8.270144462585449e-08,
-      "num_tokens": 647338.0,
-      "reward": 0.21226384192705156,
-      "reward_std": 0.06548679377883673,
-      "rewards/compute_reward/mean": 0.21226384192705156,
-      "rewards/compute_reward/std": 0.0654867960140109,
-      "step": 180,
-      "step_time": 10.853807216498534
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.9461549550294877,
-      "epoch": 0.19,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.7145590782165527,
-      "learning_rate": 2.75e-07,
-      "loss": -2.1532177925109862e-07,
-      "num_tokens": 682026.0,
-      "reward": 0.21948475018143654,
-      "reward_std": 0.05461370516568422,
-      "rewards/compute_reward/mean": 0.21948475018143654,
-      "rewards/compute_reward/std": 0.05461370553821325,
-      "step": 190,
-      "step_time": 10.456350517399551
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": 1.0,
-      "completions/max_length": 100.0,
-      "completions/max_terminated_length": 0.0,
-      "completions/mean_length": 100.0,
-      "completions/mean_terminated_length": 0.0,
-      "completions/min_length": 100.0,
-      "completions/min_terminated_length": 0.0,
-      "entropy": 0.8442220821976661,
-      "epoch": 0.2,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 3.7965171337127686,
-      "learning_rate": 2.5000000000000002e-08,
-      "loss": 1.0430812835693359e-08,
-      "num_tokens": 716746.0,
-      "reward": 0.2305009976029396,
-      "reward_std": 0.03879760131239891,
-      "rewards/compute_reward/mean": 0.2305009976029396,
-      "rewards/compute_reward/std": 0.03879760047420859,
-      "step": 200,
-      "step_time": 10.340635509999993
-    }
-  ],
-  "logging_steps": 10,
-  "max_steps": 200,
-  "num_input_tokens_seen": 716746,
-  "num_train_epochs": 1,
-  "save_steps": 100,
-  "stateful_callbacks": {
-    "TrainerControl": {
-      "args": {
-        "should_epoch_stop": false,
-        "should_evaluate": false,
-        "should_log": false,
-        "should_save": true,
-        "should_training_stop": true
-      },
-      "attributes": {}
-    }
-  },
-  "total_flos": 0.0,
-  "train_batch_size": 2,
-  "trial_name": null,
-  "trial_params": null
-}

training/checkpoints/unified_final/config.json

@@ -1,32 +0,0 @@
-{
-  "architectures": [
-    "LlamaForCausalLM"
-  ],
-  "attention_bias": false,
-  "attention_dropout": 0.0,
-  "bos_token_id": 1,
-  "dtype": "float32",
-  "eos_token_id": 2,
-  "head_dim": 64,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 5632,
-  "max_position_embeddings": 2048,
-  "mlp_bias": false,
-  "model_type": "llama",
-  "num_attention_heads": 32,
-  "num_hidden_layers": 22,
-  "num_key_value_heads": 4,
-  "pad_token_id": 2,
-  "pretraining_tp": 1,
-  "rms_norm_eps": 1e-05,
-  "rope_parameters": {
-    "rope_theta": 10000.0,
-    "rope_type": "default"
-  },
-  "tie_word_embeddings": false,
-  "transformers_version": "5.3.0",
-  "use_cache": false,
-  "vocab_size": 32000
-}

training/checkpoints/unified_final/generation_config.json

@@ -1,9 +0,0 @@
-{
-  "bos_token_id": 1,
-  "eos_token_id": [
-    2
-  ],
-  "max_length": 2048,
-  "pad_token_id": 2,
-  "transformers_version": "5.3.0"
-}

training/checkpoints/unified_final/tokenizer_config.json

@@ -1,19 +0,0 @@
-{
-  "add_prefix_space": null,
-  "backend": "tokenizers",
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "eos_token": "</s>",
-  "is_local": true,
-  "max_length": null,
-  "model_max_length": 2048,
-  "pad_to_multiple_of": null,
-  "pad_token": "</s>",
-  "pad_token_type_id": 0,
-  "padding_side": "left",
-  "sp_model_kwargs": {},
-  "tokenizer_class": "LlamaTokenizer",
-  "truncation_side": "left",
-  "unk_token": "<unk>",
-  "use_default_system_prompt": false
-}

training/checkpoints/unified_final/unified_reward_log.json

@@ -1,810 +0,0 @@
-{
-  "accuracy": [
-    0.012478123821101302,
-    0.013689774048328765,
-    0.12357050236883002,
-    0.043150096433237195,
-    0.11808098944816375,
-    0.14478551750907398,
-    0.21936089415676943,
-    0.14560732765872023,
-    0.12766012796254073,
-    0.16228250732999258,
-    0.19256023689530533,
-    0.153446869824083,
-    0.08735395734236795,
-    0.25620539761275585,
-    0.2796424323605421,
-    0.4050695781981913,
-    0.34320680785281277,
-    0.39042326634482405,
-    0.24141882976569753,
-    0.2882491476114424,
-    0.2805112680700598,
-    0.1299182187184869,
-    0.18283964773559502,
-    0.08174918994377885,
-    0.1305077084983307,
-    0.15188368799701088,
-    0.10731278214010087,
-    0.10817607256366782,
-    0.1742403849902705,
-    0.15966549523684162,
-    0.21224383614993403,
-    0.30634267989144903,
-    0.2563189622014761,
-    0.13088561721084532,
-    0.23896305011421776,
-    0.36338720554077614,
-    0.2743395734578371,
-    0.2785670698390685,
-    0.26690704237418583,
-    0.23420825800444123,
-    0.4486492634482796,
-    0.3085314377908274,
-    0.27236165767163295,
-    0.351135627192783,
-    0.37157259147763155,
-    0.4091061054548437,
-    0.3321387716436809,
-    0.25690332708634805,
-    0.4042620632377111,
-    0.21426805183517378,
-    0.46486986328175767,
-    0.5354255396266014,
-    0.5316739152617584,
-    0.3626249278251227,
-    0.5560084815324287,
-    0.47374602488847506,
-    0.5622030981309204,
-    0.6260334739834723,
-    0.5388746766273916,
-    0.43546972183358157,
-    0.4384314355118149,
-    0.43255371653260083,
-    0.382003842773009,
-    0.33916141995282467,
-    0.4102824234143368,
-    0.4002692943218704,
-    0.4433627484561765,
-    0.5707634448719365,
-    0.3326736211199734,
-    0.41868448313128437,
-    0.4830820909726724,
-    0.5073173724203757,
-    0.6011403764343056,
-    0.2652010267221505,
-    0.5708498617899997,
-    0.5372080254474398,
-    0.34268688791221447,
-    0.36077516272765764,
-    0.6577040443039563,
-    0.5249539674929385,
-    0.3393068936409599,
-    0.3981918416905377,
-    0.5998766558760262,
-    0.3886278953534839,
-    0.47030574201103836,
-    0.5933578772929455,
-    0.629797753552287,
-    0.6829957361516797,
-    0.5975855789903534,
-    0.37033629002672747,
-    0.40129960235208273,
-    0.44104763492941856,
-    0.5250475457257945,
-    0.5792574424612014,
-    0.25491493314992414,
-    0.4456432306425367,
-    0.3674802188566988,
-    0.5168529125349757,
-    0.7135775878197881,
-    0.408872426591652,
-    0.29645813006976085,
-    0.5807047440217663,
-    0.3951396545427582,
-    0.5820897600332913,
-    0.5751887943251881,
-    0.6462836385320105,
-    0.452535930180199,
-    0.6309295986678539,
-    0.521345004487674,
-    0.7523772581521466,
-    0.3868275580258203,
-    0.6621844534173644,
-    0.757102247782526,
-    0.7496667811480936,
-    0.765902349873787,
-    0.7620735178706088,
-    0.8005386810387373,
-    0.7600417191929723,
-    0.7790964529097753,
-    0.8060362095807505,
-    0.6639245812548539,
-    0.49642928937921477,
-    0.4622820479255877,
-    0.5039745619269863,
-    0.5521504355740943,
-    0.763103948879152,
-    0.3649169562800698,
-    0.8642640291197355,
-    0.7673212948914258,
-    0.6856467187291327,
-    0.6203947744628628,
-    0.635864180446877,
-    0.7076110516058842,
-    0.45257112707172986,
-    0.4927382976084982,
-    0.735338338570779,
-    0.7325108773598185,
-    0.5286115260781837,
-    0.6873601944038981,
-    0.7558585478414992,
-    0.8025525164825894,
-    0.5403924472630024,
-    0.8109585656614495,
-    0.45960476465808653,
-    0.7726514123926349,
-    0.78036072270019,
-    0.5612159043391909,
-    0.668619691132455,
-    0.7187997825397312,
-    0.6008389099901545,
-    0.5160061409523324,
-    0.6712722339255528,
-    0.25213094055121654,
-    0.7931299787283417,
-    0.5770709363152806,
-    0.3674653100689218,
-    0.7533031922202384,
-    0.5477579357220128,
-    0.9013020257140825,
-    0.774595058715597,
-    0.5444791193214735,
-    0.28536322558907645,
-    0.8018009673613502,
-    0.7534115956222964,
-    0.8178817865612724,
-    0.7691389758719754,
-    0.746364161759599,
-    0.7686015134039534,
-    0.734219302571865,
-    0.32221002464589255,
-    0.47941368112339633,
-    0.7168057798061833,
-    0.772261652825011,
-    0.5291935548529084,
-    0.7485607594114032,
-    0.5932522241567504,
-    0.5648661194163807,
-    0.5709367030781823,
-    0.7752278802176389,
-    0.6248770881515031,
-    0.5446761697530746,
-    0.8044651419608864,
-    0.855248827897706,
-    0.5436122580157401,
-    0.9085174062877894,
-    0.31500336882736524,
-    0.6913784691774245,
-    0.5400797382818436,
-    0.6050753133365693,
-    0.7986505120673587,
-    0.8202528873914283,
-    0.6996518377501237,
-    0.8313200483947909,
-    0.4808844911385792,
-    0.7306097140061414,
-    0.5058602896511918,
-    0.6438089653119033,
-    0.7879260241436392,
-    0.8337068369817564,
-    0.537435884385747
-  ],
-  "outcome": [
-    0.4,
-    0.42500000000000004,
-    0.4375,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.4,
-    0.25,
-    0.4,
-    0.0,
-    0.0,
-    0.0,
-    0.0,
-    0.07500000000000001,
-    0.025,
-    0.07500000000000001,
-    0.0,
-    0.07500000000000001,
-    0.05,
-    0.07500000000000001,
-    0.225,
-    0.4,
-    0.4,
-    0.4,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.35000000000000003,
-    0.175,
-    0.15,
-    0.15000000000000002,
-    0.07500000000000001,
-    0.17500000000000002,
-    0.1,
-    0.0,
-    0.05,
-    0.07500000000000001,
-    0.07500000000000001,
-    0.07500000000000001,
-    0.025,
-    0.0,
-    0.0,
-    0.0,
-    0.07500000000000001,
-    0.15000000000000002,
-    0.0,
-    0.05,
-    0.0,
-    0.025,
-    0.0,
-    0.0,
-    0.0,
-    0.05,
-    0.0,
-    0.05,
-    0.025,
-    0.07500000000000001,
-    0.0,
-    0.05,
-    0.025,
-    0.1,
-    0.025,
-    0.025,
-    0.025,
-    0.025,
-    0.0,
-    0.05,
-    0.05,
-    0.0,
-    0.05,
-    0.0,
-    0.0,
-    0.025,
-    0.05,
-    0.025,
-    0.0,
-    0.025,
-    0.05,
-    0.07500000000000001,
-    0.125,
-    0.25,
-    0.125,
-    0.2,
-    0.05,
-    0.17500000000000002,
-    0.225,
-    0.2,
-    0.30000000000000004,
-    0.375,
-    0.35,
-    0.42500000000000004,
-    0.35000000000000003,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.4,
-    0.42500000000000004,
-    0.42500000000000004,
-    0.45,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.45,
-    0.35000000000000003,
-    0.4,
-    0.4,
-    0.4,
-    0.35000000000000003,
-    0.4,
-    0.4,
-    0.25,
-    0.25,
-    0.35000000000000003,
-    0.4,
-    0.35000000000000003,
-    0.30000000000000004,
-    0.4,
-    0.35000000000000003,
-    0.35000000000000003,
-    0.35000000000000003,
-    0.4,
-    0.35000000000000003,
-    0.35000000000000003,
-    0.2,
-    0.35000000000000003,
-    0.4,
-    0.35000000000000003,
-    0.42500000000000004,
-    0.4,
-    0.30000000000000004,
-    0.4,
-    0.4,
-    0.42500000000000004,
-    0.42500000000000004,
-    0.4,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.35000000000000003,
-    0.42500000000000004,
-    0.30000000000000004,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.4,
-    0.42500000000000004,
-    0.4,
-    0.35000000000000003,
-    0.4,
-    0.42500000000000004,
-    0.4,
-    0.42500000000000004,
-    0.25,
-    0.35000000000000003,
-    0.4,
-    0.4,
-    0.35000000000000003,
-    0.4,
-    0.4,
-    0.35000000000000003,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.4,
-    0.375,
-    0.4,
-    0.375,
-    0.4,
-    0.35000000000000003,
-    0.4,
-    0.4,
-    0.35000000000000003,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.42500000000000004,
-    0.4,
-    0.4,
-    0.4,
-    0.4,
-    0.45,
-    0.4,
-    0.4,
-    0.4,
-    0.35000000000000003,
-    0.4,
-    0.4
-  ],
-  "bluff": [
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5,
-    -0.5
-  ],
-  "total": [
-    -0.005632656662614553,
-    0.0035414209169150612,
-    0.0463746758290905,
-    0.01385253375163301,
-    0.031328346306857296,
-    0.040674931128175884,
-    0.06677631295486929,
-    -0.011537435319447932,
-    0.03468104478688924,
-    -0.0932011224345026,
-    -0.08260391708664314,
-    -0.09629359556157094,
-    -0.11942611493017122,
-    -0.03407811083553544,
-    -0.04337514867381029,
-    0.018024352369366947,
-    -0.02987761725151553,
-    0.012898143220688411,
-    -0.04800340958200586,
-    -0.022862798335995183,
-    0.026928943824520928,
-    0.03547137655147041,
-    0.05399387670745824,
-    0.018612216480322585,
-    0.044427697974415745,
-    0.043159290798953795,
-    0.027559473749035293,
-    0.02786162539728372,
-    0.05098413474659466,
-    0.045882923332894544,
-    0.04678534265247689,
-    0.018469937962007153,
-    -0.007788363229483373,
-    -0.05169003397620414,
-    -0.04011293246002378,
-    0.03843552193927165,
-    -0.018981149289757013,
-    -0.05250152555632605,
-    -0.039082535169034954,
-    -0.04177710969844557,
-    0.033277242206897865,
-    -0.015763996773210408,
-    -0.045923419814928465,
-    -0.02710253048252593,
-    -0.019949592982828956,
-    -0.006812863090804698,
-    -0.007501429924711707,
-    -0.007583835519778186,
-    -0.008508277866801141,
-    -0.05750618185768919,
-    0.012704452148615191,
-    0.0461489388693105,
-    0.036085870341615436,
-    -0.023081275261207068,
-    0.04460296853635004,
-    0.03331110871096628,
-    0.04677108434582211,
-    0.0866117158942153,
-    0.04735613681958707,
-    0.02866440264175356,
-    0.0034510024291352186,
-    0.01889380078641028,
-    -0.00754865502944687,
-    0.0037064969834886344,
-    0.0023488481950178913,
-    -0.001155746987345354,
-    0.013926961959661782,
-    0.058517205705177766,
-    -0.03356423260800931,
-    0.014039569095949535,
-    0.03657873184043532,
-    0.02756108034713149,
-    0.07789913175200697,
-    -0.05717964064724733,
-    0.04979745162649989,
-    0.04677280890660393,
-    -0.012559589230724939,
-    -0.014978693045319853,
-    0.08019641550638473,
-    0.04248388862252848,
-    -0.01374258722566403,
-    0.015617144591688177,
-    0.10370682955660918,
-    0.07351976337371936,
-    0.05835700970386343,
-    0.12767525705253094,
-    0.08792921374330046,
-    0.1502985076530879,
-    0.13790495264662364,
-    0.049617701509354614,
-    0.09545486082322892,
-    0.13561667222529647,
-    0.15626664100402804,
-    0.2014901048614205,
-    0.06172022660247342,
-    0.15472513072488783,
-    0.11861807659984457,
-    0.1708985193872415,
-    0.23975215573692582,
-    0.1418553493070782,
-    0.10251034552441629,
-    0.21074666040761822,
-    0.12829887908996535,
-    0.19373141601165192,
-    0.19131607801381584,
-    0.21619927348620369,
-    0.1483875755630696,
-    0.2108253595337488,
-    0.18997075157068588,
-    0.23583204035325128,
-    0.12538964530903712,
-    0.22176455869607747,
-    0.25498578672388406,
-    0.2348833734018327,
-    0.25806582245582543,
-    0.256725731254713,
-    0.217688538363558,
-    0.20351460171754027,
-    0.24518375851842128,
-    0.2721126733532626,
-    0.2048736034391988,
-    0.12875025128272513,
-    0.15179871677395568,
-    0.14889109667444517,
-    0.16575265245093296,
-    0.23958638210770317,
-    0.11772093469802442,
-    0.27499241019190734,
-    0.24106245321199898,
-    0.15997635155519643,
-    0.18963817106200198,
-    0.21255246315640697,
-    0.22016386806205945,
-    0.1571498944751054,
-    0.16245840416297436,
-    0.21236841849977267,
-    0.24637880707593643,
-    0.17501403412736427,
-    0.23932606804136433,
-    0.2633004917445247,
-    0.27089338076890623,
-    0.1878873565420508,
-    0.2738354979815073,
-    0.15086166763033024,
-    0.24292799433742218,
-    0.27187625294506645,
-    0.1514255665187168,
-    0.2327668918963592,
-    0.24157992388890587,
-    0.20029361849655403,
-    0.1706021493333163,
-    0.23369528187394348,
-    0.07824582919292578,
-    0.25009549255491953,
-    0.19197482771034816,
-    0.1273628585241226,
-    0.25365611727708337,
-    0.19046527750270448,
-    0.25295570899992886,
-    0.24360827055045886,
-    0.1805676917625157,
-    0.08987712895617675,
-    0.25313033857647255,
-    0.25369405846780374,
-    0.2762586252964453,
-    0.24169864155519138,
-    0.2512274566158596,
-    0.25901052969138366,
-    0.24697675590015272,
-    0.10277350862606237,
-    0.1577947883931887,
-    0.2408820229321641,
-    0.2602915784887538,
-    0.1839677441985179,
-    0.2519962657939911,
-    0.19763827845486265,
-    0.18770314179573322,
-    0.1810778460773638,
-    0.26132975807617365,
-    0.1999569808530261,
-    0.1806366594135761,
-    0.2540627996863101,
-    0.28933708976419703,
-    0.18026429030550906,
-    0.2904810922007262,
-    0.10900117908957782,
-    0.2319824642120985,
-    0.17902790839864524,
-    0.2105263596677992,
-    0.26952767922357546,
-    0.27708851058699985,
-    0.23487814321254327,
-    0.2809620169381768,
-    0.1758095718985027,
-    0.2457133999021494,
-    0.1670511013779171,
-    0.21533313785916613,
-    0.2482741084502737,
-    0.2817973929436147,
-    0.1781025595350114
-  ]
-}

training/unified_training.log

@@ -1,269 +0,0 @@
-Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
-Loading Phase 2 checkpoint...
-
-BertModel LOAD REPORT from: sentence-transformers/all-MiniLM-L6-v2
-Key                     | Status     |  | 
-------------------------+------------+--+-
-embeddings.position_ids | UNEXPECTED |  | 
-
-Notes:
-- UNEXPECTED	:can be ignored when loading from different task/architecture; not ok if you expect identical arch.
-Building training dataset from selfplay states...
-
-Starting unified training...
-Loading from: training/checkpoints/phase2_final
-Saving to:    training/checkpoints/unified_final
-==================================================
-
  0%|          | 0/200 [00:00<?, ?it/s]Passing `generation_config` together with generation-related arguments=({'disable_compile'}) is deprecated and will be removed in future versions. Please pass either a `generation_config` object OR all generation parameters explicitly, but not both.
-Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
-
-DistilBertModel LOAD REPORT from: distilbert-base-uncased
-Key                     | Status     |  | 
-------------------------+------------+--+-
-vocab_projector.bias    | UNEXPECTED |  | 
-vocab_layer_norm.weight | UNEXPECTED |  | 
-vocab_layer_norm.bias   | UNEXPECTED |  | 
-vocab_transform.weight  | UNEXPECTED |  | 
-vocab_transform.bias    | UNEXPECTED |  | 
-
-Notes:
-- UNEXPECTED	:can be ignored when loading from different task/architecture; not ok if you expect identical arch.
-
  0%|          | 1/200 [00:14<48:31, 14.63s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  1%|          | 2/200 [00:25<40:55, 12.40s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  2%|▏         | 3/200 [00:37<40:52, 12.45s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  2%|▏         | 4/200 [00:50<41:02, 12.56s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  2%|▎         | 5/200 [01:03<40:57, 12.60s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  3%|▎         | 6/200 [01:14<39:37, 12.26s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  4%|▎         | 7/200 [01:27<39:44, 12.36s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  4%|▍         | 8/200 [01:39<38:51, 12.14s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  4%|▍         | 9/200 [01:50<37:33, 11.80s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  5%|▌         | 10/200 [02:01<36:55, 11.66s/it]
                                                
  5%|▌         | 10/200 [02:01<36:55, 11.66s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  6%|▌         | 11/200 [02:14<37:32, 11.92s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  6%|▌         | 12/200 [02:25<36:31, 11.66s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  6%|▋         | 13/200 [02:36<36:00, 11.56s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  7%|▋         | 14/200 [02:47<35:24, 11.42s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  8%|▊         | 15/200 [03:00<36:25, 11.81s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  8%|▊         | 16/200 [03:12<36:28, 11.89s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  8%|▊         | 17/200 [03:25<37:13, 12.20s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
  9%|▉         | 18/200 [03:37<36:45, 12.12s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 10%|▉         | 19/200 [03:48<36:05, 11.97s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 10%|█         | 20/200 [04:01<36:13, 12.07s/it]
                                                
 10%|█         | 20/200 [04:01<36:13, 12.07s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 10%|█         | 21/200 [04:14<36:57, 12.39s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 11%|█         | 22/200 [04:25<35:33, 11.99s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 12%|█▏        | 23/200 [04:37<35:37, 12.07s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 12%|█▏        | 24/200 [04:50<35:42, 12.18s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 12%|█▎        | 25/200 [05:00<34:15, 11.75s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 13%|█▎        | 26/200 [05:14<35:27, 12.23s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 14%|█▎        | 27/200 [05:25<34:26, 11.95s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 14%|█▍        | 28/200 [05:36<33:26, 11.66s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 14%|█▍        | 29/200 [05:49<34:30, 12.11s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 15%|█▌        | 30/200 [06:02<34:38, 12.23s/it]
                                                
 15%|█▌        | 30/200 [06:02<34:38, 12.23s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 16%|█▌        | 31/200 [06:13<33:24, 11.86s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 16%|█▌        | 32/200 [06:25<33:57, 12.13s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 16%|█▋        | 33/200 [06:38<34:16, 12.31s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 17%|█▋        | 34/200 [06:49<33:09, 11.99s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 18%|█▊        | 35/200 [07:00<32:13, 11.72s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 18%|█▊        | 36/200 [07:13<33:05, 12.11s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 18%|█▊        | 37/200 [07:24<32:02, 11.79s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 19%|█▉        | 38/200 [07:38<33:23, 12.37s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 20%|█▉        | 39/200 [07:52<34:03, 12.70s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 20%|██        | 40/200 [08:03<32:28, 12.18s/it]
                                                
 20%|██        | 40/200 [08:03<32:28, 12.18s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 20%|██        | 41/200 [08:15<32:11, 12.15s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 21%|██        | 42/200 [08:26<31:37, 12.01s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 22%|██▏       | 43/200 [08:37<30:27, 11.64s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 22%|██▏       | 44/200 [08:50<31:14, 12.01s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 22%|██▎       | 45/200 [09:03<31:47, 12.30s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 23%|██▎       | 46/200 [09:15<31:12, 12.16s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 24%|██▎       | 47/200 [09:27<30:57, 12.14s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 24%|██▍       | 48/200 [09:38<30:09, 11.90s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 24%|██▍       | 49/200 [09:52<31:30, 12.52s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 25%|██▌       | 50/200 [10:03<30:13, 12.09s/it]
                                                
 25%|██▌       | 50/200 [10:04<30:13, 12.09s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 26%|██▌       | 51/200 [10:18<31:43, 12.77s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 26%|██▌       | 52/200 [10:30<31:11, 12.64s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 26%|██▋       | 53/200 [10:42<30:30, 12.45s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 27%|██▋       | 54/200 [10:56<31:35, 12.98s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 28%|██▊       | 55/200 [11:10<31:37, 13.09s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 28%|██▊       | 56/200 [11:22<30:41, 12.79s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 28%|██▊       | 57/200 [11:33<29:33, 12.41s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 29%|██▉       | 58/200 [11:46<29:53, 12.63s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 30%|██▉       | 59/200 [11:59<29:27, 12.53s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 30%|███       | 60/200 [12:10<28:30, 12.22s/it]
                                                
 30%|███       | 60/200 [12:10<28:30, 12.22s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 30%|███       | 61/200 [12:22<27:57, 12.07s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 31%|███       | 62/200 [12:35<28:26, 12.37s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 32%|███▏      | 63/200 [12:48<28:43, 12.58s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 32%|███▏      | 64/200 [12:59<27:41, 12.21s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 32%|███▎      | 65/200 [13:12<27:59, 12.44s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 33%|███▎      | 66/200 [13:25<27:39, 12.39s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 34%|███▎      | 67/200 [13:37<27:13, 12.28s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 34%|███▍      | 68/200 [13:49<27:13, 12.37s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 34%|███▍      | 69/200 [14:00<26:03, 11.94s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 35%|███▌      | 70/200 [14:11<25:18, 11.68s/it]
                                                
 35%|███▌      | 70/200 [14:11<25:18, 11.68s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 36%|███▌      | 71/200 [14:24<25:42, 11.95s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 36%|███▌      | 72/200 [14:35<24:53, 11.67s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 36%|███▋      | 73/200 [14:48<25:25, 12.01s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 37%|███▋      | 74/200 [14:58<24:11, 11.52s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 38%|███▊      | 75/200 [15:11<24:38, 11.83s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 38%|███▊      | 76/200 [15:22<24:14, 11.73s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 38%|███▊      | 77/200 [15:33<23:19, 11.38s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 39%|███▉      | 78/200 [15:44<23:00, 11.31s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 40%|███▉      | 79/200 [15:57<23:41, 11.75s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 40%|████      | 80/200 [16:08<23:32, 11.77s/it]
                                                
 40%|████      | 80/200 [16:09<23:32, 11.77s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 40%|████      | 81/200 [16:20<23:29, 11.84s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 41%|████      | 82/200 [16:32<23:06, 11.75s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 42%|████▏     | 83/200 [16:43<22:32, 11.56s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 42%|████▏     | 84/200 [16:55<22:24, 11.59s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 42%|████▎     | 85/200 [17:06<22:05, 11.52s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 43%|████▎     | 86/200 [17:18<22:11, 11.68s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 44%|████▎     | 87/200 [17:30<22:11, 11.79s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 44%|████▍     | 88/200 [17:42<21:53, 11.73s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 44%|████▍     | 89/200 [17:53<21:22, 11.55s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 45%|████▌     | 90/200 [18:04<21:07, 11.52s/it]
                                                
 45%|████▌     | 90/200 [18:05<21:07, 11.52s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 46%|████▌     | 91/200 [18:17<21:21, 11.76s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 46%|████▌     | 92/200 [18:30<21:45, 12.09s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 46%|████▋     | 93/200 [18:41<21:18, 11.95s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 47%|████▋     | 94/200 [18:52<20:46, 11.76s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 48%|████▊     | 95/200 [19:03<20:08, 11.51s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 48%|████▊     | 96/200 [19:15<20:08, 11.62s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 48%|████▊     | 97/200 [19:28<20:36, 12.01s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 49%|████▉     | 98/200 [19:41<20:57, 12.33s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 50%|████▉     | 99/200 [19:54<21:03, 12.51s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 50%|█████     | 100/200 [20:05<20:03, 12.04s/it]
                                                 
 50%|█████     | 100/200 [20:05<20:03, 12.04s/it]{'loss': '3.099e-07', 'grad_norm': '0.7345', 'learning_rate': '4.775e-06', 'num_tokens': '3.58e+04', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.01269', 'rewards/compute_reward/std': '0.02462', 'reward': '0.01269', 'reward_std': '0.02462', 'frac_reward_zero_std': '0', 'entropy': '1.357', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.71', 'epoch': '0.01'}
-{'loss': '-4.247e-07', 'grad_norm': '4.442', 'learning_rate': '4.525e-06', 'num_tokens': '7.175e+04', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '-0.04456', 'rewards/compute_reward/std': '0.03567', 'reward': '-0.04456', 'reward_std': '0.03567', 'frac_reward_zero_std': '0', 'entropy': '1.317', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.64', 'epoch': '0.02'}
-{'loss': '-0.001816', 'grad_norm': '3.017', 'learning_rate': '4.275e-06', 'num_tokens': '1.082e+05', 'completions/mean_length': '99.86', 'completions/min_length': '98.9', 'completions/max_length': '100', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '8.9', 'completions/min_terminated_length': '8.9', 'completions/max_terminated_length': '8.9', 'rewards/compute_reward/mean': '0.03749', 'rewards/compute_reward/std': '0.02062', 'reward': '0.03749', 'reward_std': '0.02062', 'frac_reward_zero_std': '0', 'entropy': '1.006', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.76', 'epoch': '0.03'}
-{'loss': '-0.006361', 'grad_norm': '5.866', 'learning_rate': '4.025e-06', 'num_tokens': '1.434e+05', 'completions/mean_length': '99.58', 'completions/min_length': '96.6', 'completions/max_length': '100', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '6.6', 'completions/min_terminated_length': '6.6', 'completions/max_terminated_length': '6.6', 'rewards/compute_reward/mean': '-0.01482', 'rewards/compute_reward/std': '0.067', 'reward': '-0.01482', 'reward_std': '0.067', 'frac_reward_zero_std': '0', 'entropy': '1.782', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.79', 'epoch': '0.04'}
-{'loss': '-0.01103', 'grad_norm': '6.191', 'learning_rate': '3.775e-06', 'num_tokens': '1.789e+05', 'completions/mean_length': '99.12', 'completions/min_length': '93', 'completions/max_length': '100', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '3', 'completions/min_terminated_length': '3', 'completions/max_terminated_length': '3', 'rewards/compute_reward/mean': '-0.01634', 'rewards/compute_reward/std': '0.05182', 'reward': '-0.01634', 'reward_std': '0.05182', 'frac_reward_zero_std': '0', 'entropy': '2.131', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.74', 'epoch': '0.05'}
-{'loss': '0.001951', 'grad_norm': '8.536', 'learning_rate': '3.525e-06', 'num_tokens': '2.163e+05', 'completions/mean_length': '99.85', 'completions/min_length': '98.8', 'completions/max_length': '100', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '8.8', 'completions/min_terminated_length': '8.8', 'completions/max_terminated_length': '8.8', 'rewards/compute_reward/mean': '0.03592', 'rewards/compute_reward/std': '0.04931', 'reward': '0.03592', 'reward_std': '0.04931', 'frac_reward_zero_std': '0', 'entropy': '2.104', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '11.27', 'epoch': '0.06'}
-{'loss': '-0.02075', 'grad_norm': '6.919', 'learning_rate': '3.275e-06', 'num_tokens': '2.511e+05', 'completions/mean_length': '92.96', 'completions/min_length': '70.1', 'completions/max_length': '100', 'completions/clipped_ratio': '0.8', 'completions/mean_terminated_length': '38.51', 'completions/min_terminated_length': '30.1', 'completions/max_terminated_length': '48.2', 'rewards/compute_reward/mean': '0.007262', 'rewards/compute_reward/std': '0.08027', 'reward': '0.007262', 'reward_std': '0.08027', 'frac_reward_zero_std': '0', 'entropy': '1.647', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.77', 'epoch': '0.07'}
-{'loss': '0.00825', 'grad_norm': '4.918', 'learning_rate': '3.025e-06', 'num_tokens': '2.857e+05', 'completions/mean_length': '99.14', 'completions/min_length': '93.1', 'completions/max_length': '100', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '3.1', 'completions/min_terminated_length': '3.1', 'completions/max_terminated_length': '3.1', 'rewards/compute_reward/mean': '0.02766', 'rewards/compute_reward/std': '0.0484', 'reward': '0.02766', 'reward_std': '0.0484', 'frac_reward_zero_std': '0', 'entropy': '2.234', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.43', 'epoch': '0.08'}
-{'loss': '2.217e-08', 'grad_norm': '4.417', 'learning_rate': '2.775e-06', 'num_tokens': '3.202e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.07909', 'rewards/compute_reward/std': '0.07921', 'reward': '0.07909', 'reward_std': '0.07921', 'frac_reward_zero_std': '0', 'entropy': '1.806', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.34', 'epoch': '0.09'}
-{'loss': '8.494e-08', 'grad_norm': '3.353', 'learning_rate': '2.525e-06', 'num_tokens': '3.554e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.1476', 'rewards/compute_reward/std': '0.07424', 'reward': '0.1476', 'reward_std': '0.07424', 'frac_reward_zero_std': '0', 'entropy': '1.406', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.75', 'epoch': '0.1'}
-
-
-
-Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 50%|█████     | 101/200 [20:36<29:22, 17.80s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 51%|█████     | 102/200 [20:50<26:49, 16.42s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 52%|█████▏    | 103/200 [21:01<24:14, 14.99s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 52%|█████▏    | 104/200 [21:14<23:08, 14.47s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 52%|█████▎    | 105/200 [21:28<22:16, 14.06s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 53%|█████▎    | 106/200 [21:40<21:29, 13.72s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 54%|█████▎    | 107/200 [21:52<20:09, 13.00s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 54%|█████▍    | 108/200 [22:06<20:24, 13.31s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 55%|█████▍    | 109/200 [22:17<19:09, 12.63s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 55%|█████▌    | 110/200 [22:31<19:44, 13.17s/it]
                                                 
 55%|█████▌    | 110/200 [22:32<19:44, 13.17s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 56%|█████▌    | 111/200 [22:44<19:22, 13.06s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 56%|█████▌    | 112/200 [22:56<18:50, 12.85s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 56%|█████▋    | 113/200 [23:10<18:49, 12.98s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 57%|█████▋    | 114/200 [23:21<18:04, 12.61s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 57%|█████▊    | 115/200 [23:34<17:45, 12.54s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 58%|█████▊    | 116/200 [23:46<17:21, 12.40s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 58%|█████▊    | 117/200 [23:59<17:25, 12.59s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 59%|█████▉    | 118/200 [24:13<17:38, 12.91s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 60%|█████▉    | 119/200 [24:25<17:23, 12.89s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 60%|██████    | 120/200 [24:38<16:57, 12.71s/it]
                                                 
 60%|██████    | 120/200 [24:38<16:57, 12.71s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 60%|██████    | 121/200 [24:52<17:12, 13.06s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 61%|██████    | 122/200 [25:04<16:44, 12.88s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 62%|██████▏   | 123/200 [25:14<15:30, 12.08s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 62%|██████▏   | 124/200 [25:27<15:30, 12.25s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 62%|██████▎   | 125/200 [25:40<15:28, 12.38s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 63%|██████▎   | 126/200 [25:52<15:14, 12.36s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 64%|██████▎   | 127/200 [26:03<14:31, 11.94s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 64%|██████▍   | 128/200 [26:17<15:03, 12.55s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 64%|██████▍   | 129/200 [26:29<14:45, 12.47s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 65%|██████▌   | 130/200 [26:42<14:45, 12.64s/it]
                                                 
 65%|██████▌   | 130/200 [26:42<14:45, 12.64s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 66%|██████▌   | 131/200 [26:54<14:14, 12.38s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 66%|██████▌   | 132/200 [27:05<13:30, 11.92s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 66%|██████▋   | 133/200 [27:16<13:12, 11.83s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 67%|██████▋   | 134/200 [27:28<12:46, 11.61s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 68%|██████▊   | 135/200 [27:39<12:32, 11.58s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 68%|██████▊   | 136/200 [27:51<12:37, 11.83s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 68%|██████▊   | 137/200 [28:04<12:32, 11.95s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 69%|██████▉   | 138/200 [28:18<13:00, 12.59s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 70%|██████▉   | 139/200 [28:31<12:56, 12.73s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 70%|███████   | 140/200 [28:44<12:51, 12.86s/it]
                                                 
 70%|███████   | 140/200 [28:44<12:51, 12.86s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 70%|███████   | 141/200 [28:56<12:23, 12.60s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 71%|███████   | 142/200 [29:08<11:56, 12.36s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 72%|███████▏  | 143/200 [29:20<11:38, 12.26s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 72%|███████▏  | 144/200 [29:31<11:01, 11.81s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 72%|███████▎  | 145/200 [29:43<10:59, 11.99s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 73%|███████▎  | 146/200 [29:56<11:09, 12.40s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 74%|███████▎  | 147/200 [30:10<11:22, 12.88s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 74%|███████▍  | 148/200 [30:22<10:51, 12.53s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 74%|███████▍  | 149/200 [30:34<10:23, 12.22s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 75%|███████▌  | 150/200 [30:46<10:13, 12.27s/it]
                                                 
 75%|███████▌  | 150/200 [30:46<10:13, 12.27s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 76%|███████▌  | 151/200 [30:57<09:49, 12.03s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 76%|███████▌  | 152/200 [31:09<09:33, 11.96s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 76%|███████▋  | 153/200 [31:20<09:04, 11.58s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 77%|███████▋  | 154/200 [31:32<09:02, 11.80s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 78%|███████▊  | 155/200 [31:44<08:56, 11.91s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 78%|███████▊  | 156/200 [31:55<08:32, 11.65s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 78%|███████▊  | 157/200 [32:08<08:35, 11.98s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 79%|███████▉  | 158/200 [32:20<08:18, 11.87s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 80%|███████▉  | 159/200 [32:33<08:25, 12.32s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 80%|████████  | 160/200 [32:45<08:10, 12.25s/it]
                                                 
 80%|████████  | 160/200 [32:45<08:10, 12.25s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 80%|████████  | 161/200 [32:56<07:40, 11.80s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 81%|████████  | 162/200 [33:07<07:19, 11.58s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 82%|████████▏ | 163/200 [33:20<07:25, 12.03s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 82%|████████▏ | 164/200 [33:33<07:24, 12.35s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 82%|████████▎ | 165/200 [33:47<07:25, 12.74s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 83%|████████▎ | 166/200 [33:59<07:10, 12.68s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 84%|████████▎ | 167/200 [34:13<07:02, 12.80s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 84%|████████▍ | 168/200 [34:24<06:36, 12.39s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 84%|████████▍ | 169/200 [34:37<06:28, 12.52s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 85%|████████▌ | 170/200 [34:48<06:01, 12.06s/it]
                                                 
 85%|████████▌ | 170/200 [34:48<06:01, 12.06s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 86%|████████▌ | 171/200 [34:59<05:41, 11.76s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 86%|████████▌ | 172/200 [35:11<05:29, 11.77s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 86%|████████▋ | 173/200 [35:22<05:14, 11.65s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 87%|████████▋ | 174/200 [35:34<05:03, 11.66s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 88%|████████▊ | 175/200 [35:45<04:52, 11.71s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 88%|████████▊ | 176/200 [35:59<04:50, 12.11s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 88%|████████▊ | 177/200 [36:12<04:45, 12.42s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 89%|████████▉ | 178/200 [36:24<04:33, 12.43s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 90%|████████▉ | 179/200 [36:37<04:21, 12.45s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 90%|█████████ | 180/200 [36:50<04:12, 12.63s/it]
                                                 
 90%|█████████ | 180/200 [36:50<04:12, 12.63s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 90%|█████████ | 181/200 [37:01<03:51, 12.19s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 91%|█████████ | 182/200 [37:15<03:47, 12.66s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 92%|█████████▏| 183/200 [37:27<03:33, 12.56s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 92%|█████████▏| 184/200 [37:39<03:17, 12.34s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 92%|█████████▎| 185/200 [37:51<03:04, 12.29s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 93%|█████████▎| 186/200 [38:02<02:45, 11.82s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 94%|█████████▎| 187/200 [38:13<02:31, 11.62s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 94%|█████████▍| 188/200 [38:24<02:17, 11.49s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 94%|█████████▍| 189/200 [38:35<02:05, 11.40s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 95%|█████████▌| 190/200 [38:47<01:55, 11.53s/it]
                                                 
 95%|█████████▌| 190/200 [38:47<01:55, 11.53s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 96%|█████████▌| 191/200 [38:58<01:43, 11.49s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 96%|█████████▌| 192/200 [39:10<01:31, 11.39s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 96%|█████████▋| 193/200 [39:23<01:23, 11.99s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 97%|█████████▋| 194/200 [39:34<01:10, 11.79s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 98%|█████████▊| 195/200 [39:46<00:59, 11.85s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 98%|█████████▊| 196/200 [39:57<00:46, 11.55s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 98%|█████████▊| 197/200 [40:08<00:34, 11.34s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
 99%|█████████▉| 198/200 [40:20<00:23, 11.56s/it]Both `max_new_tokens` (=100) and `max_length`(=2048) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
-
-
                                                 
-{'loss': '1.505e-07', 'grad_norm': '2.707', 'learning_rate': '2.025e-06', 'num_tokens': '4.305e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.229', 'rewards/compute_reward/std': '0.0403', 'reward': '0.229', 'reward_std': '0.0403', 'frac_reward_zero_std': '0', 'entropy': '0.8805', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '11.24', 'epoch': '0.12'}
-{'loss': '1.222e-07', 'grad_norm': '3.943', 'learning_rate': '1.775e-06', 'num_tokens': '4.672e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.1833', 'rewards/compute_reward/std': '0.07255', 'reward': '0.1833', 'reward_std': '0.07255', 'frac_reward_zero_std': '0', 'entropy': '0.8755', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '11.07', 'epoch': '0.13'}
-{'loss': '-1.401e-07', 'grad_norm': '4.041', 'learning_rate': '1.525e-06', 'num_tokens': '5.03e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.2078', 'rewards/compute_reward/std': '0.06581', 'reward': '0.2078', 'reward_std': '0.06581', 'frac_reward_zero_std': '0', 'entropy': '0.9737', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.84', 'epoch': '0.14'}
-{'loss': '2.086e-08', 'grad_norm': '3.721', 'learning_rate': '1.275e-06', 'num_tokens': '5.398e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.2224', 'rewards/compute_reward/std': '0.05879', 'reward': '0.2224', 'reward_std': '0.05879', 'frac_reward_zero_std': '0', 'entropy': '0.9901', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.85', 'epoch': '0.15'}
-{'loss': '1.46e-07', 'grad_norm': '3.453', 'learning_rate': '1.025e-06', 'num_tokens': '5.754e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.1993', 'rewards/compute_reward/std': '0.06031', 'reward': '0.1993', 'reward_std': '0.06031', 'frac_reward_zero_std': '0', 'entropy': '1.121', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.62', 'epoch': '0.16'}
-{'loss': '0.0005226', 'grad_norm': '3.998', 'learning_rate': '7.75e-07', 'num_tokens': '6.12e+05', 'completions/mean_length': '99.81', 'completions/min_length': '98.5', 'completions/max_length': '100', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '8.5', 'completions/min_terminated_length': '8.5', 'completions/max_terminated_length': '8.5', 'rewards/compute_reward/mean': '0.2155', 'rewards/compute_reward/std': '0.03223', 'reward': '0.2155', 'reward_std': '0.03223', 'frac_reward_zero_std': '0', 'entropy': '0.9432', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.9', 'epoch': '0.17'}
-{'loss': '-8.27e-08', 'grad_norm': '3.733', 'learning_rate': '5.25e-07', 'num_tokens': '6.473e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.2123', 'rewards/compute_reward/std': '0.06549', 'reward': '0.2123', 'reward_std': '0.06549', 'frac_reward_zero_std': '0', 'entropy': '0.9799', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.85', 'epoch': '0.18'}
-{'loss': '-2.153e-07', 'grad_norm': '3.715', 'learning_rate': '2.75e-07', 'num_tokens': '6.82e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.2195', 'rewards/compute_reward/std': '0.05461', 'reward': '0.2195', 'reward_std': '0.05461', 'frac_reward_zero_std': '0', 'entropy': '0.9462', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.46', 'epoch': '0.19'}
-{'loss': '1.043e-08', 'grad_norm': '3.797', 'learning_rate': '2.5e-08', 'num_tokens': '7.167e+05', 'completions/mean_length': '100', 'completions/min_length': '100', 'completions/max_length': '100', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/compute_reward/mean': '0.2305', 'rewards/compute_reward/std': '0.0388', 'reward': '0.2305', 'reward_std': '0.0388', 'frac_reward_zero_std': '0', 'entropy': '0.8442', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '10.34', 'epoch': '0.2'}
-
-
-
-
                                                 
-{'train_runtime': '2458', 'train_samples_per_second': '0.651', 'train_steps_per_second': '0.081', 'train_loss': '-0.001462', 'epoch': '0.2'}
-
-Unified model saved to training/checkpoints/unified_final
-Reward curve saved to training/unified_reward_curve.png
-
-Final reward values (last 20 steps):
-  accuracy: 0.7212
-  outcome:  0.3800
-  bluff:    -0.5000
-  total:    0.2354
-
-Unified training complete.

wandb/settings

@@ -0,0 +1,3 @@
+[default]
+mode = disabled
+