127 lines
5.4 KiB
Markdown
127 lines
5.4 KiB
Markdown
# Session Log — 2026-04-19
|
|
|
|
## Key Discovery: Why Multi-Track Training Fails
|
|
|
|
### The Problem
|
|
Our multi-track training uses `close_and_switch()` which:
|
|
1. Closes the TCP connection to the sim
|
|
2. Sends `exit_scene` to go back to menu
|
|
3. Opens a NEW connection on a different track
|
|
4. Calls `model.set_env(new_env)` to swap the environment
|
|
|
|
This disrupts PPO's training because:
|
|
- PPO's rollout buffer contains partial experience from the old track
|
|
- The value function estimates become wrong for the new track
|
|
- The advantage calculations (which drive PPO's policy updates) are corrupted
|
|
- Every switch is like ripping out a student's notebook mid-lesson
|
|
|
|
### Evidence
|
|
- **Wave 4:** 25 trials with this methodology. Only 4/25 (16%) scored >500.
|
|
Median score 111. Trial 9 scored 1435 but was a lucky outlier.
|
|
- **Exp 10:** Same code, nearly identical hyperparameters to Trial 9.
|
|
Total failure — crashes on all tracks at <180 steps.
|
|
- **Conclusion:** Trial 9's success was random weight initialization luck,
|
|
not evidence the method works.
|
|
|
|
### The Fix: Parallel Environments (DummyVecEnv)
|
|
|
|
SB3's `DummyVecEnv` can wrap multiple gym environments. PPO collects
|
|
experience from ALL environments in every rollout batch. No switching,
|
|
no closing, no disruption.
|
|
|
|
```python
|
|
env = DummyVecEnv([
|
|
lambda: wrap_env(gym.make('donkey-generated-track-v0', conf={"port": 9091})),
|
|
lambda: wrap_env(gym.make('donkey-mountain-track-v0', conf={"port": 9093})),
|
|
])
|
|
env = VecTransposeImage(env)
|
|
model = PPO('CnnPolicy', env, ...)
|
|
model.learn(total_timesteps=90000) # both tracks in EVERY batch
|
|
```
|
|
|
|
This requires two sim instances on different ports (one track per sim),
|
|
but gives PPO a stable, consistent training setup — exactly how SB3 is
|
|
designed to work with multiple environments.
|
|
|
|
### How DummyVecEnv Works (for future reference)
|
|
|
|
PPO training loop (simplified):
|
|
```
|
|
for each rollout batch:
|
|
for each of N steps in rollout:
|
|
for each env in DummyVecEnv: ← env[0]=generated_track, env[1]=mountain_track
|
|
action = policy(observation)
|
|
next_obs, reward, done = env.step(action)
|
|
store (obs, action, reward, done) in buffer
|
|
|
|
compute advantages using value function
|
|
update policy using all experience from ALL envs
|
|
```
|
|
|
|
Key insight: the model doesn't "know" which track it's on. It just sees
|
|
images and learns a policy that works across all the images it sees.
|
|
Both tracks contribute to every policy update. This prevents catastrophic
|
|
forgetting because the model never stops seeing either track.
|
|
|
|
With close_and_switch: model trains on track A for 6000 steps, completely
|
|
forgets track A while training on track B for 6000 steps, etc. Classic
|
|
catastrophic interference.
|
|
|
|
With DummyVecEnv: model sees both tracks simultaneously in every batch.
|
|
Like a human alternating laps between two courses — never forgets either one.
|
|
|
|
### Alternative: Same Env, Switch Track Scene
|
|
|
|
Theoretically possible: keep TCP connection open, send `exit_scene` then
|
|
`load_scene(new_track)` without closing the gym env. The observation and
|
|
action spaces are identical across tracks so SB3 wouldn't notice.
|
|
|
|
Concerns:
|
|
- gym_donkeycar's DonkeyEnv initializes scene in __init__, not designed
|
|
for mid-session scene changes
|
|
- The viewer/sim controller state machine may not handle re-loading cleanly
|
|
- Still sequential (not parallel) so still has the forgetting problem,
|
|
just without the env close/reopen disruption
|
|
- Untested — could introduce subtle bugs
|
|
|
|
### Hardware Options
|
|
- Two sim instances on same machine (different ports: 9091, 9093)
|
|
- Risk: GPU memory pressure from two Unity instances
|
|
- Second sim on remote machine
|
|
- gym_donkeycar supports `host` parameter in conf
|
|
- Previous connection issues to remote host need debugging
|
|
|
|
### Image Augmentation (complementary, not primary)
|
|
DonkeyCar sim has built-in augmentation options:
|
|
- Gaussian blur, image flipping, cropping
|
|
- Other donkeycar users use these for generalization
|
|
- Solves visual robustness (lighting, noise) but NOT track geometry diversity
|
|
- Best used TOGETHER with parallel multi-track training
|
|
|
|
### Warm Start Failure Re-Analysis
|
|
Previously tried warm-starting from generated_road champion onto multi-track
|
|
training. This failed — but it used the broken close_and_switch methodology.
|
|
The warm start itself may not have been the problem. Worth retrying once
|
|
parallel envs are working.
|
|
|
|
## Exp 10 Evaluation Results (re-run 2026-04-19)
|
|
|
|
| Track | Set 1 | Set 2 | Set 3 | Mean | Verdict |
|
|
|---|---|---|---|---|---|
|
|
| mountain_track (trained) | 178 | 179 | 179 | **179** | ❌ Crashes at same spot |
|
|
| generated_track (trained) | 99 | 82 | 88 | **90** | ❌ Crashes immediately |
|
|
| generated_road (zero-shot) | 135 | 223 | 105 | **154** | ❌ Crashes early |
|
|
| mini_monaco (zero-shot) | 111 | 133 | 129 | **124** | ❌ Crashes early |
|
|
|
|
## Next Steps
|
|
- **Exp 11:** Tested parallel DummyVecEnv with two sim instances (ports 9091 + 9093)
|
|
- Exp 11 (v5 reward): aborted due to circular driving on generated_track
|
|
- Exp 11b (v6 reward): completed, no circles, but plateaus at ~194 steps on all tracks
|
|
- **v6 reward confirmed:** efficiency gate prevents circles, tests pass
|
|
- **Parallel env confirmed:** mechanically sound, stable training
|
|
- **Open issue:** 90k steps may be insufficient for 2-env training (45k per track)
|
|
- **Next experiment ideas:**
|
|
- Increase to 180k-250k total steps
|
|
- Test v6 on single track to isolate reward effect
|
|
- Check if efficiency gate fires during normal cornering (false positives)
|