chore(exp24): launch exp24, fix logging setup, update handoff

- Exp23 complete (mean 2000 steps / 408.6r, high variance confirms nose-first stuck issue)
- Unity 6000.4.4f1 rebuild done: Assembly-CSharp.dll updated with Car.cs raycast fix
- Rsync'd to runtime folder, sim restarted on port 9091
- Exp24 launched (PID 733053) — discrete(7), speed stuck, road regen
- Fix logging.basicConfig no-op: use file_log.addHandler() directly
- Monitor via /tmp/exp24.out (log file was 0 bytes with old approach)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul Huliganga 2026-05-05 20:12:36 -04:00
parent 78d81827b7
commit c6a18e7fee
2 changed files with 47 additions and 35 deletions

View File

@ -1,6 +1,6 @@
# RL Donkeycar Session Handoff
Last updated: 2026-05-05 America/Toronto (updated during exp23)
Last updated: 2026-05-05 20:09 America/Toronto (updated during exp24 launch)
## Autonomy Instruction
@ -12,11 +12,12 @@ If the user says only `continue`, interpret it using the instruction above.
## Current Goal
Run exp24 on `generated_road` with:
- Discrete steering (7 bins) — smoother, less oscillation than continuous
- Speed-based stuck detection — catches car pressed nose-first against barrier
- Road regeneration — sim reconnects between segments so each eval is a fresh road
- Unity raycast fix — Car.cs detects nose-first barrier contact via forward raycast
Monitor exp24 on `generated_road`. All setup complete and running.
- **Exp24 is RUNNING** (PID 733053) since 20:09 on 2026-05-05
- Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
- All fixes are in effect: discrete steering (7 bins), speed-based stuck detection,
road regeneration, Unity raycast fix compiled into Assembly-CSharp.dll (20:05)
## Important Paths
@ -101,29 +102,29 @@ to Unity physics non-determinism, NOT road variety.
## Current State
### Exp 23 status
- Running (PID 649531), at ~140k/200k steps as of last check
- Will finish on its own — DO NOT kill it
- Log: `agent/models/exp23-generated-road-clean/run_2026-05-05_160718_clean.log`
### Exp 23 status — COMPLETE
- Finished at 18:12 on 2026-05-05
- Final mean: 2000 steps / 408.6 reward
- High variance throughout: some crashes in 105171 steps (nose-first barrier),
others ran 16841951 steps. Best eval: 403.8r / 2000s ✅ at ~40k steps.
- This confirmed the nose-first stuck issue that exp24 is designed to fix.
### Unity build status
- **Needs rebuild** — Car.cs raycast fix not yet compiled
- Car.cs was modified at:
`/mnt/c/Users/Paul/Documents/projects/sdsandbox/sdsim/Assets/Scripts/Car.cs`
### Unity build status — DONE
- Rebuilt with Unity 6000.4.4f1 at ~20:05 on 2026-05-05
- Assembly-CSharp.dll updated: includes Car.cs forward raycast fix
- Rsync'd to: `/mnt/c/Users/Paul/Downloads/DonkeySimWin/DonkeySimWin/`
- Sim restarted on port 9091 with new binary
### To launch exp24
1. Wait for exp23 to finish (or confirm it has)
2. Rebuild Unity (Car.cs raycast fix)
3. Stop sim on port 9091
4. Rsync build to runtime folders (both, or just the one on 9091)
5. Restart sim on port 9091
6. Launch exp24:
### Exp 24 status — RUNNING
- PID 733053, launched at 20:09 on 2026-05-05
- Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
- Monitor with:
```bash
cd /home/paulh/projects/donkeycar-rl-autoresearch/agent
nohup python3 experiments/exp24_generated_road_discrete.py > /tmp/exp24.out 2>&1 &
tail -f /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp24-discrete/run_*_discrete.log
tail -f /tmp/exp24.out
```
NOTE: The log file (run_*_discrete.log) is 0 bytes — logging.basicConfig() is a
no-op because gym_donkeycar already configured the root logger. All output goes
to /tmp/exp24.out via nohup. Fix in exp25: use `file_log.addHandler(FileHandler(...))` directly.
## Useful Commands
@ -169,13 +170,19 @@ grep "Eval\|BEST" agent/models/exp23-generated-road-clean/run_2026-05-05_160718_
## Notes for Next Session
- Unity rebuild is required before exp24 — Car.cs raycast fix won't be in effect
until the build is done and synced.
- Unity rebuild is DONE. No rebuild needed unless Car.cs or other scripts change.
- Assembly-CSharp.dll confirmed in runtime folder. Port 9091 sim uses new binary.
- The second sim (port 9093) is not needed — only port 9091.
- Do NOT kill exp23 — let it run to completion.
- Exp24 is running. DO NOT kill it. Let it run to 200k steps.
- Exp24's road regeneration adds ~5s per checkpoint = ~100s extra total. This is
by design. The "Reconnecting for fresh road" log lines confirm it's working.
- `info['speed']` from telemetry = `rb.velocity.magnitude / 8.0`. The
`LOW_SPEED_THRESHOLD=0.5` corresponds to 4 Unity m/s, which is slow but not zero.
A truly stuck car reads ~0.0. Tight corners might temporarily be 0.10.3.
The 2-second timer provides enough grace for normal slow driving.
- Unity version used: 6000.4.4f1 (NOT 2020.3.x — the project is on Unity 6).
Build command: `"/mnt/c/Program Files/Unity/Hub/Editor/6000.4.4f1/Editor/Unity.exe"
-quit -batchmode -projectPath "C:/Users/Paul/Documents/projects/sdsandbox/sdsim"
-executeMethod PlayerBuilder.WinBuild -logFile "C:/Users/Paul/AppData/Local/Temp/unity_rebuild.log"`
- To kill/restart sim from WSL2: `taskkill.exe` is at `/mnt/c/Windows/System32/taskkill.exe`
(not in PATH — use full path). Restart: `/mnt/c/Windows/System32/cmd.exe /c start "" "C:\...\donkey_sim.exe" --port 9091`

View File

@ -168,12 +168,17 @@ log_path = os.path.join(_SAVE_DIR, f'run_{run_tag}.log')
best_model_path = os.path.join(_SAVE_DIR, 'best_model.zip')
import logging
logging.basicConfig(
level=logging.INFO,
format='%(message)s',
handlers=[logging.FileHandler(log_path), logging.StreamHandler(sys.stdout)],
)
# logging.basicConfig is a no-op if the root logger already has handlers (e.g. from
# gym_donkeycar/SB3 imports). Add handlers directly to avoid silent file-log loss.
_file_handler = logging.FileHandler(log_path)
_file_handler.setFormatter(logging.Formatter('%(message)s'))
_stream_handler = logging.StreamHandler(sys.stdout)
_stream_handler.setFormatter(logging.Formatter('%(message)s'))
file_log = logging.getLogger('exp24')
file_log.setLevel(logging.INFO)
file_log.propagate = False
file_log.addHandler(_file_handler)
file_log.addHandler(_stream_handler)
def flog(msg):