chore(exp24): launch exp24, fix logging setup, update handoff

- Exp23 complete (mean 2000 steps / 408.6r, high variance confirms nose-first stuck issue)
- Unity 6000.4.4f1 rebuild done: Assembly-CSharp.dll updated with Car.cs raycast fix
- Rsync'd to runtime folder, sim restarted on port 9091
- Exp24 launched (PID 733053) — discrete(7), speed stuck, road regen
- Fix logging.basicConfig no-op: use file_log.addHandler() directly
- Monitor via /tmp/exp24.out (log file was 0 bytes with old approach)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul Huliganga 2026-05-05 20:12:36 -04:00
parent 78d81827b7
commit c6a18e7fee
2 changed files with 47 additions and 35 deletions

View File

@ -1,6 +1,6 @@
# RL Donkeycar Session Handoff # RL Donkeycar Session Handoff
Last updated: 2026-05-05 America/Toronto (updated during exp23) Last updated: 2026-05-05 20:09 America/Toronto (updated during exp24 launch)
## Autonomy Instruction ## Autonomy Instruction
@ -12,11 +12,12 @@ If the user says only `continue`, interpret it using the instruction above.
## Current Goal ## Current Goal
Run exp24 on `generated_road` with: Monitor exp24 on `generated_road`. All setup complete and running.
- Discrete steering (7 bins) — smoother, less oscillation than continuous
- Speed-based stuck detection — catches car pressed nose-first against barrier - **Exp24 is RUNNING** (PID 733053) since 20:09 on 2026-05-05
- Road regeneration — sim reconnects between segments so each eval is a fresh road - Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
- Unity raycast fix — Car.cs detects nose-first barrier contact via forward raycast - All fixes are in effect: discrete steering (7 bins), speed-based stuck detection,
road regeneration, Unity raycast fix compiled into Assembly-CSharp.dll (20:05)
## Important Paths ## Important Paths
@ -101,29 +102,29 @@ to Unity physics non-determinism, NOT road variety.
## Current State ## Current State
### Exp 23 status ### Exp 23 status — COMPLETE
- Running (PID 649531), at ~140k/200k steps as of last check - Finished at 18:12 on 2026-05-05
- Will finish on its own — DO NOT kill it - Final mean: 2000 steps / 408.6 reward
- Log: `agent/models/exp23-generated-road-clean/run_2026-05-05_160718_clean.log` - High variance throughout: some crashes in 105171 steps (nose-first barrier),
others ran 16841951 steps. Best eval: 403.8r / 2000s ✅ at ~40k steps.
- This confirmed the nose-first stuck issue that exp24 is designed to fix.
### Unity build status ### Unity build status — DONE
- **Needs rebuild** — Car.cs raycast fix not yet compiled - Rebuilt with Unity 6000.4.4f1 at ~20:05 on 2026-05-05
- Car.cs was modified at: - Assembly-CSharp.dll updated: includes Car.cs forward raycast fix
`/mnt/c/Users/Paul/Documents/projects/sdsandbox/sdsim/Assets/Scripts/Car.cs` - Rsync'd to: `/mnt/c/Users/Paul/Downloads/DonkeySimWin/DonkeySimWin/`
- Sim restarted on port 9091 with new binary
### To launch exp24 ### Exp 24 status — RUNNING
- PID 733053, launched at 20:09 on 2026-05-05
1. Wait for exp23 to finish (or confirm it has) - Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
2. Rebuild Unity (Car.cs raycast fix) - Monitor with:
3. Stop sim on port 9091
4. Rsync build to runtime folders (both, or just the one on 9091)
5. Restart sim on port 9091
6. Launch exp24:
```bash ```bash
cd /home/paulh/projects/donkeycar-rl-autoresearch/agent tail -f /tmp/exp24.out
nohup python3 experiments/exp24_generated_road_discrete.py > /tmp/exp24.out 2>&1 &
tail -f /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp24-discrete/run_*_discrete.log
``` ```
NOTE: The log file (run_*_discrete.log) is 0 bytes — logging.basicConfig() is a
no-op because gym_donkeycar already configured the root logger. All output goes
to /tmp/exp24.out via nohup. Fix in exp25: use `file_log.addHandler(FileHandler(...))` directly.
## Useful Commands ## Useful Commands
@ -169,13 +170,19 @@ grep "Eval\|BEST" agent/models/exp23-generated-road-clean/run_2026-05-05_160718_
## Notes for Next Session ## Notes for Next Session
- Unity rebuild is required before exp24 — Car.cs raycast fix won't be in effect - Unity rebuild is DONE. No rebuild needed unless Car.cs or other scripts change.
until the build is done and synced. - Assembly-CSharp.dll confirmed in runtime folder. Port 9091 sim uses new binary.
- The second sim (port 9093) is not needed — only port 9091. - The second sim (port 9093) is not needed — only port 9091.
- Do NOT kill exp23 — let it run to completion. - Exp24 is running. DO NOT kill it. Let it run to 200k steps.
- Exp24's road regeneration adds ~5s per checkpoint = ~100s extra total. This is - Exp24's road regeneration adds ~5s per checkpoint = ~100s extra total. This is
by design. The "Reconnecting for fresh road" log lines confirm it's working. by design. The "Reconnecting for fresh road" log lines confirm it's working.
- `info['speed']` from telemetry = `rb.velocity.magnitude / 8.0`. The - `info['speed']` from telemetry = `rb.velocity.magnitude / 8.0`. The
`LOW_SPEED_THRESHOLD=0.5` corresponds to 4 Unity m/s, which is slow but not zero. `LOW_SPEED_THRESHOLD=0.5` corresponds to 4 Unity m/s, which is slow but not zero.
A truly stuck car reads ~0.0. Tight corners might temporarily be 0.10.3. A truly stuck car reads ~0.0. Tight corners might temporarily be 0.10.3.
The 2-second timer provides enough grace for normal slow driving. The 2-second timer provides enough grace for normal slow driving.
- Unity version used: 6000.4.4f1 (NOT 2020.3.x — the project is on Unity 6).
Build command: `"/mnt/c/Program Files/Unity/Hub/Editor/6000.4.4f1/Editor/Unity.exe"
-quit -batchmode -projectPath "C:/Users/Paul/Documents/projects/sdsandbox/sdsim"
-executeMethod PlayerBuilder.WinBuild -logFile "C:/Users/Paul/AppData/Local/Temp/unity_rebuild.log"`
- To kill/restart sim from WSL2: `taskkill.exe` is at `/mnt/c/Windows/System32/taskkill.exe`
(not in PATH — use full path). Restart: `/mnt/c/Windows/System32/cmd.exe /c start "" "C:\...\donkey_sim.exe" --port 9091`

View File

@ -168,12 +168,17 @@ log_path = os.path.join(_SAVE_DIR, f'run_{run_tag}.log')
best_model_path = os.path.join(_SAVE_DIR, 'best_model.zip') best_model_path = os.path.join(_SAVE_DIR, 'best_model.zip')
import logging import logging
logging.basicConfig( # logging.basicConfig is a no-op if the root logger already has handlers (e.g. from
level=logging.INFO, # gym_donkeycar/SB3 imports). Add handlers directly to avoid silent file-log loss.
format='%(message)s', _file_handler = logging.FileHandler(log_path)
handlers=[logging.FileHandler(log_path), logging.StreamHandler(sys.stdout)], _file_handler.setFormatter(logging.Formatter('%(message)s'))
) _stream_handler = logging.StreamHandler(sys.stdout)
_stream_handler.setFormatter(logging.Formatter('%(message)s'))
file_log = logging.getLogger('exp24') file_log = logging.getLogger('exp24')
file_log.setLevel(logging.INFO)
file_log.propagate = False
file_log.addHandler(_file_handler)
file_log.addHandler(_stream_handler)
def flog(msg): def flog(msg):