122 lines
12 KiB
Plaintext
122 lines
12 KiB
Plaintext
[2026-04-14 22:40:44] =================================================================
|
||
[2026-04-14 22:40:44] [Wave4] Multi-Track Autoresearch — GP+UCB Generalization Search
|
||
[2026-04-14 22:40:44] [Wave4] Training tracks : generated_track, mountain_track (no generated_road, no warm-start)
|
||
[2026-04-14 22:40:44] [Wave4] Test tracks : mini_monaco only (zero-shot; warren removed — broken done condition)
|
||
[2026-04-14 22:40:44] [Wave4] Max trials : 25 | kappa=2.0 | push every 5
|
||
[2026-04-14 22:40:44] [Wave4] Results file : /home/paulh/projects/donkeycar-rl-autoresearch/agent/outerloop-results/autoresearch_results_phase4.jsonl
|
||
[2026-04-14 22:40:44] [Wave4] Champion dir : /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-champion
|
||
[2026-04-14 22:40:44] [Wave4] Warm start : NONE (training from scratch each trial)
|
||
[2026-04-14 22:40:44] =================================================================
|
||
[2026-04-14 22:40:44] [Wave4] Loaded 0 existing Phase 3 results.
|
||
[2026-04-14 22:40:44] [Wave4] No Wave 3 champion yet.
|
||
[2026-04-14 22:40:44] [Wave4] Starting from trial 1.
|
||
[2026-04-14 22:40:44]
|
||
[Wave4] ========== Trial 1/25 ==========
|
||
[2026-04-14 22:40:44] [Wave4] Seed trial 1/2: using hardcoded params.
|
||
[2026-04-14 22:40:44] [Wave4] Proposed params: {'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-14 22:40:46] [Wave4] Launching trial 1: {'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-14 22:40:46] [Wave4] Command: python3 /home/paulh/projects/donkeycar-rl-autoresearch/agent/multitrack_runner.py --total-timesteps 80000 --steps-per-switch 6000 --learning-rate 0.0003 --eval-episodes 3 --save-dir /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-trial-0001
|
||
[2026-04-14 22:44:24] =================================================================
|
||
[2026-04-14 22:44:24] [Wave4] Multi-Track Autoresearch — GP+UCB Generalization Search
|
||
[2026-04-14 22:44:24] [Wave4] Training tracks : generated_track, mountain_track (no generated_road, no warm-start)
|
||
[2026-04-14 22:44:24] [Wave4] Test tracks : mini_monaco only (zero-shot; warren removed — broken done condition)
|
||
[2026-04-14 22:44:24] [Wave4] Max trials : 25 | kappa=2.0 | push every 5
|
||
[2026-04-14 22:44:24] [Wave4] Results file : /home/paulh/projects/donkeycar-rl-autoresearch/agent/outerloop-results/autoresearch_results_phase4.jsonl
|
||
[2026-04-14 22:44:24] [Wave4] Champion dir : /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-champion
|
||
[2026-04-14 22:44:24] [Wave4] Warm start : NONE (training from scratch each trial)
|
||
[2026-04-14 22:44:24] =================================================================
|
||
[2026-04-14 22:44:24] [Wave4] Loaded 0 existing Phase 3 results.
|
||
[2026-04-14 22:44:24] [Wave4] No Wave 3 champion yet.
|
||
[2026-04-14 22:44:24] [Wave4] Starting from trial 1.
|
||
[2026-04-14 22:44:24]
|
||
[Wave4] ========== Trial 1/25 ==========
|
||
[2026-04-14 22:44:24] [Wave4] Seed trial 1/2: using hardcoded params.
|
||
[2026-04-14 22:44:24] [Wave4] Proposed params: {'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-14 22:44:26] [Wave4] Launching trial 1: {'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-14 22:44:26] [Wave4] Command: python3 /home/paulh/projects/donkeycar-rl-autoresearch/agent/multitrack_runner.py --total-timesteps 80000 --steps-per-switch 6000 --learning-rate 0.0003 --eval-episodes 3 --save-dir /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-trial-0001
|
||
[2026-04-15 00:02:45] [Wave4] Trial 1 finished in 4699.3s, rc=0
|
||
[2026-04-15 00:02:45] [Wave4] Parsed: combined=45.6693 mini_monaco=45.6693
|
||
[2026-04-15 00:02:45] [Champion] 🏆 NEW BEST! Trial 1: score=45.67 (mini_monaco=45.7) params={'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 00:02:45] [Wave4] ===== Trial 1 Summary =====
|
||
[2026-04-15 00:02:45] GP data points : 1
|
||
[2026-04-15 00:02:45] Wave4 Champion: trial=1 score=45.67 params={'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 00:02:45] Top 5:
|
||
[2026-04-15 00:02:45] score=45.67 params={'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 00:02:47]
|
||
[Wave4] ========== Trial 2/25 ==========
|
||
[2026-04-15 00:02:47] [Wave4] Seed trial 2/2: using hardcoded params.
|
||
[2026-04-15 00:02:47] [Wave4] Proposed params: {'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 00:02:49] [Wave4] Launching trial 2: {'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 00:02:49] [Wave4] Command: python3 /home/paulh/projects/donkeycar-rl-autoresearch/agent/multitrack_runner.py --total-timesteps 80000 --steps-per-switch 6000 --learning-rate 0.001 --eval-episodes 3 --save-dir /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-trial-0002
|
||
[2026-04-15 01:21:38] [Wave4] Trial 2 finished in 4728.4s, rc=0
|
||
[2026-04-15 01:21:38] [Wave4] Parsed: combined=222.0731 mini_monaco=222.0731
|
||
[2026-04-15 01:21:38] [Champion] 🏆 NEW BEST! Trial 2: score=222.07 (mini_monaco=222.1) params={'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 01:21:38] [Wave4] ===== Trial 2 Summary =====
|
||
[2026-04-15 01:21:38] GP data points : 2
|
||
[2026-04-15 01:21:38] Wave4 Champion: trial=2 score=222.07 params={'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 01:21:38] Top 5:
|
||
[2026-04-15 01:21:38] score=222.07 params={'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 01:21:38] score=45.67 params={'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 01:21:40]
|
||
[Wave4] ========== Trial 3/25 ==========
|
||
[2026-04-15 01:21:40] [Wave4] Only 2 results — using random proposal.
|
||
[2026-04-15 01:21:40] [Wave4] Proposed params: {'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 01:21:42] [Wave4] Launching trial 3: {'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 01:21:42] [Wave4] Command: python3 /home/paulh/projects/donkeycar-rl-autoresearch/agent/multitrack_runner.py --total-timesteps 157743 --steps-per-switch 17499 --learning-rate 0.0006852550685205609 --eval-episodes 3 --save-dir /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-trial-0003
|
||
[2026-04-15 03:15:46] [Wave4] Trial 3 finished in 6843.7s, rc=0
|
||
[2026-04-15 03:15:46] [Wave4] Parsed: combined=1943.1038 mini_monaco=1943.1038
|
||
[2026-04-15 03:15:46] [Champion] 🏆 NEW BEST! Trial 3: score=1943.10 (mini_monaco=1943.1) params={'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 03:15:46] [Wave4] ===== Trial 3 Summary =====
|
||
[2026-04-15 03:15:46] GP data points : 3
|
||
[2026-04-15 03:15:46] Wave4 Champion: trial=3 score=1943.10 params={'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 03:15:46] Top 5:
|
||
[2026-04-15 03:15:46] score=1943.10 params={'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 03:15:46] score=222.07 params={'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 03:15:46] score=45.67 params={'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 03:15:48]
|
||
[Wave4] ========== Trial 4/25 ==========
|
||
[2026-04-15 03:15:48] [Wave4] GP UCB top-5 proposals:
|
||
[2026-04-15 03:15:48] UCB=2.4560 mu=0.8788 σ=0.7886 params={'learning_rate': 0.0003250095463348546, 'steps_per_switch': 19054, 'total_timesteps': 197116}
|
||
[2026-04-15 03:15:48] UCB=2.4518 mu=0.8393 σ=0.8062 params={'learning_rate': 0.00121703003154963, 'steps_per_switch': 16951, 'total_timesteps': 180865}
|
||
[2026-04-15 03:15:48] UCB=2.4512 mu=0.7637 σ=0.8437 params={'learning_rate': 0.00036067077082995895, 'steps_per_switch': 16532, 'total_timesteps': 211219}
|
||
[2026-04-15 03:15:48] UCB=2.4501 mu=0.9283 σ=0.7609 params={'learning_rate': 0.0005325315186424085, 'steps_per_switch': 18992, 'total_timesteps': 205595}
|
||
[2026-04-15 03:15:48] UCB=2.4492 mu=0.9106 σ=0.7693 params={'learning_rate': 0.001163360064352729, 'steps_per_switch': 19652, 'total_timesteps': 151744}
|
||
[2026-04-15 03:15:48] [Wave4] Proposed params: {'learning_rate': 0.0003250095463348546, 'steps_per_switch': 19054, 'total_timesteps': 197116}
|
||
[2026-04-15 03:15:50] [Wave4] Launching trial 4: {'learning_rate': 0.0003250095463348546, 'steps_per_switch': 19054, 'total_timesteps': 197116}
|
||
[2026-04-15 03:15:50] [Wave4] Command: python3 /home/paulh/projects/donkeycar-rl-autoresearch/agent/multitrack_runner.py --total-timesteps 197116 --steps-per-switch 19054 --learning-rate 0.0003250095463348546 --eval-episodes 3 --save-dir /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-trial-0004
|
||
[2026-04-15 05:15:51] [Wave4] Trial 4 TIMED OUT — killing runner.
|
||
[2026-04-15 05:15:51] [Wave4] Trial 4 finished in 7200.5s, rc=-9
|
||
[2026-04-15 05:15:51] [Wave4] Parsed: combined=None mini_monaco=None
|
||
[2026-04-15 05:15:51] [Wave4] ⚠️ No test score parsed — defaulting to 0.0
|
||
[2026-04-15 05:15:51] [Wave4] combined_test_score=0 — excluded from GP (crash/timeout).
|
||
[2026-04-15 05:15:51] [Wave4] ===== Trial 4 Summary =====
|
||
[2026-04-15 05:15:51] GP data points : 3
|
||
[2026-04-15 05:15:51] Wave4 Champion: trial=3 score=1943.10 params={'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 05:15:51] Top 5:
|
||
[2026-04-15 05:15:51] score=1943.10 params={'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 05:15:51] score=222.07 params={'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 05:15:51] score=45.67 params={'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 05:15:53]
|
||
[Wave4] ========== Trial 5/25 ==========
|
||
[2026-04-15 05:15:53] [Wave4] GP UCB top-5 proposals:
|
||
[2026-04-15 05:15:53] UCB=2.4597 mu=0.8329 σ=0.8134 params={'learning_rate': 0.0003927960467617446, 'steps_per_switch': 19892, 'total_timesteps': 201785}
|
||
[2026-04-15 05:15:53] UCB=2.4568 mu=0.8585 σ=0.7991 params={'learning_rate': 0.0011330710879806035, 'steps_per_switch': 18089, 'total_timesteps': 193054}
|
||
[2026-04-15 05:15:53] UCB=2.4560 mu=0.7832 σ=0.8364 params={'learning_rate': 0.0006110661120319741, 'steps_per_switch': 17141, 'total_timesteps': 219583}
|
||
[2026-04-15 05:15:53] UCB=2.4560 mu=0.8338 σ=0.8111 params={'learning_rate': 0.000602366907571214, 'steps_per_switch': 16527, 'total_timesteps': 215069}
|
||
[2026-04-15 05:15:53] UCB=2.4522 mu=0.8120 σ=0.8201 params={'learning_rate': 0.0004035684210100053, 'steps_per_switch': 16067, 'total_timesteps': 208387}
|
||
[2026-04-15 05:15:53] [Wave4] Proposed params: {'learning_rate': 0.0003927960467617446, 'steps_per_switch': 19892, 'total_timesteps': 201785}
|
||
[2026-04-15 05:15:55] [Wave4] Launching trial 5: {'learning_rate': 0.0003927960467617446, 'steps_per_switch': 19892, 'total_timesteps': 201785}
|
||
[2026-04-15 05:15:55] [Wave4] Command: python3 /home/paulh/projects/donkeycar-rl-autoresearch/agent/multitrack_runner.py --total-timesteps 201785 --steps-per-switch 19892 --learning-rate 0.0003927960467617446 --eval-episodes 3 --save-dir /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/wave4-trial-0005
|
||
[2026-04-15 07:15:57] [Wave4] Trial 5 TIMED OUT — killing runner.
|
||
[2026-04-15 07:15:57] [Wave4] Trial 5 finished in 7202.3s, rc=-9
|
||
[2026-04-15 07:15:57] [Wave4] Parsed: combined=None mini_monaco=None
|
||
[2026-04-15 07:15:57] [Wave4] ⚠️ No test score parsed — defaulting to 0.0
|
||
[2026-04-15 07:15:57] [Wave4] combined_test_score=0 — excluded from GP (crash/timeout).
|
||
[2026-04-15 07:15:57] [Wave4] ===== Trial 5 Summary =====
|
||
[2026-04-15 07:15:57] GP data points : 3
|
||
[2026-04-15 07:15:57] Wave4 Champion: trial=3 score=1943.10 params={'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 07:15:57] Top 5:
|
||
[2026-04-15 07:15:57] score=1943.10 params={'learning_rate': 0.0006852550685205609, 'steps_per_switch': 17499, 'total_timesteps': 157743}
|
||
[2026-04-15 07:15:57] score=222.07 params={'learning_rate': 0.001, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|
||
[2026-04-15 07:15:57] score=45.67 params={'learning_rate': 0.0003, 'steps_per_switch': 6000, 'total_timesteps': 80000}
|