paulh
/
donkeycar-rl-autoresearch
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Research-grade DonkeyCar RL autoresearch and sweep system.
24 Commits 1 Branch 0 Tags 207 MiB
Python 94.3%
Shell 5.7%
Go to file
Paul Huliganga c8a495dd22 fix: reward v4 — full sim bypass kills circular driving at root 
ROOT CAUSE:
  donkey_sim.py calc_reward() uses forward_vel = dot(heading, velocity).
  A spinning car ALWAYS has forward_vel > 0 (always moving 'forward' relative
  to its own heading), so it earned positive reward indefinitely while circling.

v3 WAS INSUFFICIENT:
  v3 applied efficiency only to the speed BONUS: original × (1 + speed×eff×scale)
  But 'original' from sim was still exploitable: CTE≈0 while spinning → original=1.0/step
  Efficiency killed the speed bonus but not the base reward.
  47k-step run: spinning = 1.0/step × 47k = 47k reward (never crashes in circle)

v4 FIX — base × efficiency × speed:
  reward = (1 - abs(cte)/max_cte) × efficiency × (1 + speed_scale × speed)
  Completely ignores sim's bogus forward_vel reward.
  Spinning (eff≈0): reward ≈ 0 regardless of CTE or speed.
  ALL three terms must be high to earn reward — cannot be gamed.

Key new test: test_circling_at_zero_cte_gives_near_zero_reward
  Worst-case exploit (CTE=0 spinning) → avg reward < 0.15 (was 1.0 in v3)
  forward_beats_circling_by_3x confirmed.

Also: update Phase 2 autoresearch timesteps test, research log updated.

Agent: pi/claude-sonnet
Tests: 40/40 passing
Tests-Added: +1 (core v4 circling guarantee)
TypeScript: N/A
 		2026-04-13 20:56:32 -04:00
.harness feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
agent fix: reward v4 — full sim bypass kills circular driving at root 2026-04-13 20:56:32 -04:00
docs fix: reward v4 — full sim bypass kills circular driving at root 2026-04-13 20:56:32 -04:00
tests fix: reward v4 — full sim bypass kills circular driving at root 2026-04-13 20:56:32 -04:00
.gitignore feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
AGENT.md feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
DECISIONS.md feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
IMPLEMENTATION_PLAN.md feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
PROJECT-KICKOFF.md feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
PROJECT-SPEC.md feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
README.md feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00
create_gitea_repo.py Initial commit 2026-04-12 23:44:36 -04:00
ralph-loop.sh feat: Wave 1 complete — real PPO training, model save, GP+UCB autoresearch, 37 tests passing 2026-04-13 10:03:15 -04:00

README.md

donkeycar-rl-autoresearch

Purpose

Status

  • Scaffolded with the agent harness
  • Spec not filled yet

Runbook

  • Fill PROJECT-SPEC.md
  • Create IMPLEMENTATION_PLAN.md from the spec
  • Start the implementation loop