chore(exp24): launch exp24, fix logging setup, update handoff
- Exp23 complete (mean 2000 steps / 408.6r, high variance confirms nose-first stuck issue) - Unity 6000.4.4f1 rebuild done: Assembly-CSharp.dll updated with Car.cs raycast fix - Rsync'd to runtime folder, sim restarted on port 9091 - Exp24 launched (PID 733053) — discrete(7), speed stuck, road regen - Fix logging.basicConfig no-op: use file_log.addHandler() directly - Monitor via /tmp/exp24.out (log file was 0 bytes with old approach) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
78d81827b7
commit
c6a18e7fee
|
|
@ -1,6 +1,6 @@
|
|||
# RL Donkeycar Session Handoff
|
||||
|
||||
Last updated: 2026-05-05 America/Toronto (updated during exp23)
|
||||
Last updated: 2026-05-05 20:09 America/Toronto (updated during exp24 launch)
|
||||
|
||||
## Autonomy Instruction
|
||||
|
||||
|
|
@ -12,11 +12,12 @@ If the user says only `continue`, interpret it using the instruction above.
|
|||
|
||||
## Current Goal
|
||||
|
||||
Run exp24 on `generated_road` with:
|
||||
- Discrete steering (7 bins) — smoother, less oscillation than continuous
|
||||
- Speed-based stuck detection — catches car pressed nose-first against barrier
|
||||
- Road regeneration — sim reconnects between segments so each eval is a fresh road
|
||||
- Unity raycast fix — Car.cs detects nose-first barrier contact via forward raycast
|
||||
Monitor exp24 on `generated_road`. All setup complete and running.
|
||||
|
||||
- **Exp24 is RUNNING** (PID 733053) since 20:09 on 2026-05-05
|
||||
- Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
|
||||
- All fixes are in effect: discrete steering (7 bins), speed-based stuck detection,
|
||||
road regeneration, Unity raycast fix compiled into Assembly-CSharp.dll (20:05)
|
||||
|
||||
## Important Paths
|
||||
|
||||
|
|
@ -101,29 +102,29 @@ to Unity physics non-determinism, NOT road variety.
|
|||
|
||||
## Current State
|
||||
|
||||
### Exp 23 status
|
||||
- Running (PID 649531), at ~140k/200k steps as of last check
|
||||
- Will finish on its own — DO NOT kill it
|
||||
- Log: `agent/models/exp23-generated-road-clean/run_2026-05-05_160718_clean.log`
|
||||
### Exp 23 status — COMPLETE
|
||||
- Finished at 18:12 on 2026-05-05
|
||||
- Final mean: 2000 steps / 408.6 reward
|
||||
- High variance throughout: some crashes in 105–171 steps (nose-first barrier),
|
||||
others ran 1684–1951 steps. Best eval: 403.8r / 2000s ✅ at ~40k steps.
|
||||
- This confirmed the nose-first stuck issue that exp24 is designed to fix.
|
||||
|
||||
### Unity build status
|
||||
- **Needs rebuild** — Car.cs raycast fix not yet compiled
|
||||
- Car.cs was modified at:
|
||||
`/mnt/c/Users/Paul/Documents/projects/sdsandbox/sdsim/Assets/Scripts/Car.cs`
|
||||
### Unity build status — DONE
|
||||
- Rebuilt with Unity 6000.4.4f1 at ~20:05 on 2026-05-05
|
||||
- Assembly-CSharp.dll updated: includes Car.cs forward raycast fix
|
||||
- Rsync'd to: `/mnt/c/Users/Paul/Downloads/DonkeySimWin/DonkeySimWin/`
|
||||
- Sim restarted on port 9091 with new binary
|
||||
|
||||
### To launch exp24
|
||||
|
||||
1. Wait for exp23 to finish (or confirm it has)
|
||||
2. Rebuild Unity (Car.cs raycast fix)
|
||||
3. Stop sim on port 9091
|
||||
4. Rsync build to runtime folders (both, or just the one on 9091)
|
||||
5. Restart sim on port 9091
|
||||
6. Launch exp24:
|
||||
### Exp 24 status — RUNNING
|
||||
- PID 733053, launched at 20:09 on 2026-05-05
|
||||
- Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
|
||||
- Monitor with:
|
||||
```bash
|
||||
cd /home/paulh/projects/donkeycar-rl-autoresearch/agent
|
||||
nohup python3 experiments/exp24_generated_road_discrete.py > /tmp/exp24.out 2>&1 &
|
||||
tail -f /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp24-discrete/run_*_discrete.log
|
||||
tail -f /tmp/exp24.out
|
||||
```
|
||||
NOTE: The log file (run_*_discrete.log) is 0 bytes — logging.basicConfig() is a
|
||||
no-op because gym_donkeycar already configured the root logger. All output goes
|
||||
to /tmp/exp24.out via nohup. Fix in exp25: use `file_log.addHandler(FileHandler(...))` directly.
|
||||
|
||||
## Useful Commands
|
||||
|
||||
|
|
@ -169,13 +170,19 @@ grep "Eval\|BEST" agent/models/exp23-generated-road-clean/run_2026-05-05_160718_
|
|||
|
||||
## Notes for Next Session
|
||||
|
||||
- Unity rebuild is required before exp24 — Car.cs raycast fix won't be in effect
|
||||
until the build is done and synced.
|
||||
- Unity rebuild is DONE. No rebuild needed unless Car.cs or other scripts change.
|
||||
- Assembly-CSharp.dll confirmed in runtime folder. Port 9091 sim uses new binary.
|
||||
- The second sim (port 9093) is not needed — only port 9091.
|
||||
- Do NOT kill exp23 — let it run to completion.
|
||||
- Exp24 is running. DO NOT kill it. Let it run to 200k steps.
|
||||
- Exp24's road regeneration adds ~5s per checkpoint = ~100s extra total. This is
|
||||
by design. The "Reconnecting for fresh road" log lines confirm it's working.
|
||||
- `info['speed']` from telemetry = `rb.velocity.magnitude / 8.0`. The
|
||||
`LOW_SPEED_THRESHOLD=0.5` corresponds to 4 Unity m/s, which is slow but not zero.
|
||||
A truly stuck car reads ~0.0. Tight corners might temporarily be 0.1–0.3.
|
||||
The 2-second timer provides enough grace for normal slow driving.
|
||||
- Unity version used: 6000.4.4f1 (NOT 2020.3.x — the project is on Unity 6).
|
||||
Build command: `"/mnt/c/Program Files/Unity/Hub/Editor/6000.4.4f1/Editor/Unity.exe"
|
||||
-quit -batchmode -projectPath "C:/Users/Paul/Documents/projects/sdsandbox/sdsim"
|
||||
-executeMethod PlayerBuilder.WinBuild -logFile "C:/Users/Paul/AppData/Local/Temp/unity_rebuild.log"`
|
||||
- To kill/restart sim from WSL2: `taskkill.exe` is at `/mnt/c/Windows/System32/taskkill.exe`
|
||||
(not in PATH — use full path). Restart: `/mnt/c/Windows/System32/cmd.exe /c start "" "C:\...\donkey_sim.exe" --port 9091`
|
||||
|
|
|
|||
|
|
@ -168,12 +168,17 @@ log_path = os.path.join(_SAVE_DIR, f'run_{run_tag}.log')
|
|||
best_model_path = os.path.join(_SAVE_DIR, 'best_model.zip')
|
||||
|
||||
import logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(message)s',
|
||||
handlers=[logging.FileHandler(log_path), logging.StreamHandler(sys.stdout)],
|
||||
)
|
||||
# logging.basicConfig is a no-op if the root logger already has handlers (e.g. from
|
||||
# gym_donkeycar/SB3 imports). Add handlers directly to avoid silent file-log loss.
|
||||
_file_handler = logging.FileHandler(log_path)
|
||||
_file_handler.setFormatter(logging.Formatter('%(message)s'))
|
||||
_stream_handler = logging.StreamHandler(sys.stdout)
|
||||
_stream_handler.setFormatter(logging.Formatter('%(message)s'))
|
||||
file_log = logging.getLogger('exp24')
|
||||
file_log.setLevel(logging.INFO)
|
||||
file_log.propagate = False
|
||||
file_log.addHandler(_file_handler)
|
||||
file_log.addHandler(_stream_handler)
|
||||
|
||||
|
||||
def flog(msg):
|
||||
|
|
|
|||
Loading…
Reference in New Issue