chore(exp24): launch exp24, fix logging setup, update handoff
- Exp23 complete (mean 2000 steps / 408.6r, high variance confirms nose-first stuck issue) - Unity 6000.4.4f1 rebuild done: Assembly-CSharp.dll updated with Car.cs raycast fix - Rsync'd to runtime folder, sim restarted on port 9091 - Exp24 launched (PID 733053) — discrete(7), speed stuck, road regen - Fix logging.basicConfig no-op: use file_log.addHandler() directly - Monitor via /tmp/exp24.out (log file was 0 bytes with old approach) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
78d81827b7
commit
c6a18e7fee
|
|
@ -1,6 +1,6 @@
|
||||||
# RL Donkeycar Session Handoff
|
# RL Donkeycar Session Handoff
|
||||||
|
|
||||||
Last updated: 2026-05-05 America/Toronto (updated during exp23)
|
Last updated: 2026-05-05 20:09 America/Toronto (updated during exp24 launch)
|
||||||
|
|
||||||
## Autonomy Instruction
|
## Autonomy Instruction
|
||||||
|
|
||||||
|
|
@ -12,11 +12,12 @@ If the user says only `continue`, interpret it using the instruction above.
|
||||||
|
|
||||||
## Current Goal
|
## Current Goal
|
||||||
|
|
||||||
Run exp24 on `generated_road` with:
|
Monitor exp24 on `generated_road`. All setup complete and running.
|
||||||
- Discrete steering (7 bins) — smoother, less oscillation than continuous
|
|
||||||
- Speed-based stuck detection — catches car pressed nose-first against barrier
|
- **Exp24 is RUNNING** (PID 733053) since 20:09 on 2026-05-05
|
||||||
- Road regeneration — sim reconnects between segments so each eval is a fresh road
|
- Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
|
||||||
- Unity raycast fix — Car.cs detects nose-first barrier contact via forward raycast
|
- All fixes are in effect: discrete steering (7 bins), speed-based stuck detection,
|
||||||
|
road regeneration, Unity raycast fix compiled into Assembly-CSharp.dll (20:05)
|
||||||
|
|
||||||
## Important Paths
|
## Important Paths
|
||||||
|
|
||||||
|
|
@ -101,29 +102,29 @@ to Unity physics non-determinism, NOT road variety.
|
||||||
|
|
||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
### Exp 23 status
|
### Exp 23 status — COMPLETE
|
||||||
- Running (PID 649531), at ~140k/200k steps as of last check
|
- Finished at 18:12 on 2026-05-05
|
||||||
- Will finish on its own — DO NOT kill it
|
- Final mean: 2000 steps / 408.6 reward
|
||||||
- Log: `agent/models/exp23-generated-road-clean/run_2026-05-05_160718_clean.log`
|
- High variance throughout: some crashes in 105–171 steps (nose-first barrier),
|
||||||
|
others ran 1684–1951 steps. Best eval: 403.8r / 2000s ✅ at ~40k steps.
|
||||||
|
- This confirmed the nose-first stuck issue that exp24 is designed to fix.
|
||||||
|
|
||||||
### Unity build status
|
### Unity build status — DONE
|
||||||
- **Needs rebuild** — Car.cs raycast fix not yet compiled
|
- Rebuilt with Unity 6000.4.4f1 at ~20:05 on 2026-05-05
|
||||||
- Car.cs was modified at:
|
- Assembly-CSharp.dll updated: includes Car.cs forward raycast fix
|
||||||
`/mnt/c/Users/Paul/Documents/projects/sdsandbox/sdsim/Assets/Scripts/Car.cs`
|
- Rsync'd to: `/mnt/c/Users/Paul/Downloads/DonkeySimWin/DonkeySimWin/`
|
||||||
|
- Sim restarted on port 9091 with new binary
|
||||||
|
|
||||||
### To launch exp24
|
### Exp 24 status — RUNNING
|
||||||
|
- PID 733053, launched at 20:09 on 2026-05-05
|
||||||
1. Wait for exp23 to finish (or confirm it has)
|
- Log: `agent/models/exp24-discrete/run_2026-05-05_200903_discrete.log`
|
||||||
2. Rebuild Unity (Car.cs raycast fix)
|
- Monitor with:
|
||||||
3. Stop sim on port 9091
|
```bash
|
||||||
4. Rsync build to runtime folders (both, or just the one on 9091)
|
tail -f /tmp/exp24.out
|
||||||
5. Restart sim on port 9091
|
```
|
||||||
6. Launch exp24:
|
NOTE: The log file (run_*_discrete.log) is 0 bytes — logging.basicConfig() is a
|
||||||
```bash
|
no-op because gym_donkeycar already configured the root logger. All output goes
|
||||||
cd /home/paulh/projects/donkeycar-rl-autoresearch/agent
|
to /tmp/exp24.out via nohup. Fix in exp25: use `file_log.addHandler(FileHandler(...))` directly.
|
||||||
nohup python3 experiments/exp24_generated_road_discrete.py > /tmp/exp24.out 2>&1 &
|
|
||||||
tail -f /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp24-discrete/run_*_discrete.log
|
|
||||||
```
|
|
||||||
|
|
||||||
## Useful Commands
|
## Useful Commands
|
||||||
|
|
||||||
|
|
@ -169,13 +170,19 @@ grep "Eval\|BEST" agent/models/exp23-generated-road-clean/run_2026-05-05_160718_
|
||||||
|
|
||||||
## Notes for Next Session
|
## Notes for Next Session
|
||||||
|
|
||||||
- Unity rebuild is required before exp24 — Car.cs raycast fix won't be in effect
|
- Unity rebuild is DONE. No rebuild needed unless Car.cs or other scripts change.
|
||||||
until the build is done and synced.
|
- Assembly-CSharp.dll confirmed in runtime folder. Port 9091 sim uses new binary.
|
||||||
- The second sim (port 9093) is not needed — only port 9091.
|
- The second sim (port 9093) is not needed — only port 9091.
|
||||||
- Do NOT kill exp23 — let it run to completion.
|
- Exp24 is running. DO NOT kill it. Let it run to 200k steps.
|
||||||
- Exp24's road regeneration adds ~5s per checkpoint = ~100s extra total. This is
|
- Exp24's road regeneration adds ~5s per checkpoint = ~100s extra total. This is
|
||||||
by design. The "Reconnecting for fresh road" log lines confirm it's working.
|
by design. The "Reconnecting for fresh road" log lines confirm it's working.
|
||||||
- `info['speed']` from telemetry = `rb.velocity.magnitude / 8.0`. The
|
- `info['speed']` from telemetry = `rb.velocity.magnitude / 8.0`. The
|
||||||
`LOW_SPEED_THRESHOLD=0.5` corresponds to 4 Unity m/s, which is slow but not zero.
|
`LOW_SPEED_THRESHOLD=0.5` corresponds to 4 Unity m/s, which is slow but not zero.
|
||||||
A truly stuck car reads ~0.0. Tight corners might temporarily be 0.1–0.3.
|
A truly stuck car reads ~0.0. Tight corners might temporarily be 0.1–0.3.
|
||||||
The 2-second timer provides enough grace for normal slow driving.
|
The 2-second timer provides enough grace for normal slow driving.
|
||||||
|
- Unity version used: 6000.4.4f1 (NOT 2020.3.x — the project is on Unity 6).
|
||||||
|
Build command: `"/mnt/c/Program Files/Unity/Hub/Editor/6000.4.4f1/Editor/Unity.exe"
|
||||||
|
-quit -batchmode -projectPath "C:/Users/Paul/Documents/projects/sdsandbox/sdsim"
|
||||||
|
-executeMethod PlayerBuilder.WinBuild -logFile "C:/Users/Paul/AppData/Local/Temp/unity_rebuild.log"`
|
||||||
|
- To kill/restart sim from WSL2: `taskkill.exe` is at `/mnt/c/Windows/System32/taskkill.exe`
|
||||||
|
(not in PATH — use full path). Restart: `/mnt/c/Windows/System32/cmd.exe /c start "" "C:\...\donkey_sim.exe" --port 9091`
|
||||||
|
|
|
||||||
|
|
@ -168,12 +168,17 @@ log_path = os.path.join(_SAVE_DIR, f'run_{run_tag}.log')
|
||||||
best_model_path = os.path.join(_SAVE_DIR, 'best_model.zip')
|
best_model_path = os.path.join(_SAVE_DIR, 'best_model.zip')
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
logging.basicConfig(
|
# logging.basicConfig is a no-op if the root logger already has handlers (e.g. from
|
||||||
level=logging.INFO,
|
# gym_donkeycar/SB3 imports). Add handlers directly to avoid silent file-log loss.
|
||||||
format='%(message)s',
|
_file_handler = logging.FileHandler(log_path)
|
||||||
handlers=[logging.FileHandler(log_path), logging.StreamHandler(sys.stdout)],
|
_file_handler.setFormatter(logging.Formatter('%(message)s'))
|
||||||
)
|
_stream_handler = logging.StreamHandler(sys.stdout)
|
||||||
|
_stream_handler.setFormatter(logging.Formatter('%(message)s'))
|
||||||
file_log = logging.getLogger('exp24')
|
file_log = logging.getLogger('exp24')
|
||||||
|
file_log.setLevel(logging.INFO)
|
||||||
|
file_log.propagate = False
|
||||||
|
file_log.addHandler(_file_handler)
|
||||||
|
file_log.addHandler(_stream_handler)
|
||||||
|
|
||||||
|
|
||||||
def flog(msg):
|
def flog(msg):
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue