136 lines
8.7 KiB
Plaintext
136 lines
8.7 KiB
Plaintext
[07:36:52] ============================================================
|
|
[07:36:52] Exp 26 started — PID 962156
|
|
[07:36:52] Log: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/run_2026-05-06_073652_warmstart.log
|
|
[07:36:52] Warm start: exp25 best_model (381r @ 80k)
|
|
[07:36:52] ============================================================
|
|
[07:41:49] [10,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0010000.zip
|
|
[07:41:49] Reconnecting for fresh road...
|
|
[07:41:54] Connected (new road)
|
|
[07:42:20] Eval: gen_road=380.0r/2000s ✅
|
|
[07:42:20] NEW BEST: steps=2000 reward=380.0
|
|
[07:47:11] [20,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0020000.zip
|
|
[07:47:11] Reconnecting for fresh road...
|
|
[07:47:16] Connected (new road)
|
|
[07:47:42] Eval: gen_road=388.8r/2000s ✅
|
|
[07:47:43] NEW BEST: steps=2000 reward=388.8
|
|
[07:52:32] [30,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0030000.zip
|
|
[07:52:32] Reconnecting for fresh road...
|
|
[07:52:37] Connected (new road)
|
|
[07:53:03] Eval: gen_road=379.7r/2000s ✅
|
|
[07:57:55] [40,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0040000.zip
|
|
[07:57:55] Reconnecting for fresh road...
|
|
[07:58:00] Connected (new road)
|
|
[07:58:27] Eval: gen_road=371.8r/2000s ✅
|
|
[08:03:15] [50,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0050000.zip
|
|
[08:03:15] Reconnecting for fresh road...
|
|
[08:03:20] Connected (new road)
|
|
[08:03:47] Eval: gen_road=374.4r/2000s ✅
|
|
[08:08:44] [60,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0060000.zip
|
|
[08:08:44] Reconnecting for fresh road...
|
|
[08:08:50] Connected (new road)
|
|
[08:09:16] Eval: gen_road=370.9r/2000s ✅
|
|
[08:14:11] [70,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0070000.zip
|
|
[08:14:11] Reconnecting for fresh road...
|
|
[08:14:16] Connected (new road)
|
|
[08:14:42] Eval: gen_road=375.3r/2000s ✅
|
|
[08:19:33] [80,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0080000.zip
|
|
[08:19:33] Reconnecting for fresh road...
|
|
[08:19:38] Connected (new road)
|
|
[08:20:04] Eval: gen_road=359.1r/2000s ✅
|
|
[08:24:55] [90,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0090000.zip
|
|
[08:24:55] Reconnecting for fresh road...
|
|
[08:25:00] Connected (new road)
|
|
[08:25:27] Eval: gen_road=357.5r/2000s ✅
|
|
[08:30:19] [100,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0100000.zip
|
|
[08:30:19] Reconnecting for fresh road...
|
|
[08:30:24] Connected (new road)
|
|
[08:30:51] Eval: gen_road=361.5r/2000s ✅
|
|
[08:35:39] [110,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0110000.zip
|
|
[08:35:39] Reconnecting for fresh road...
|
|
[08:35:44] Connected (new road)
|
|
[08:36:10] Eval: gen_road=353.5r/2000s ✅
|
|
[08:41:01] [120,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0120000.zip
|
|
[08:41:01] Reconnecting for fresh road...
|
|
[08:41:06] Connected (new road)
|
|
[08:41:32] Eval: gen_road=355.0r/2000s ✅
|
|
[08:46:25] [130,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0130000.zip
|
|
[08:46:25] Reconnecting for fresh road...
|
|
[08:46:30] Connected (new road)
|
|
[08:46:56] Eval: gen_road=348.6r/2000s ✅
|
|
[08:51:52] [140,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0140000.zip
|
|
[08:51:52] Reconnecting for fresh road...
|
|
[08:51:57] Connected (new road)
|
|
[08:52:23] Eval: gen_road=333.6r/2000s ✅
|
|
[08:57:15] [150,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0150000.zip
|
|
[08:57:15] Reconnecting for fresh road...
|
|
[08:57:20] Connected (new road)
|
|
[08:57:46] Eval: gen_road=327.4r/2000s ✅
|
|
[09:02:40] [160,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0160000.zip
|
|
[09:02:40] Reconnecting for fresh road...
|
|
[09:02:45] Connected (new road)
|
|
[09:03:11] Eval: gen_road=332.3r/2000s ✅
|
|
[09:08:07] [170,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0170000.zip
|
|
[09:08:07] Reconnecting for fresh road...
|
|
[09:08:12] Connected (new road)
|
|
[09:08:22] Eval: gen_road=96.7r/626s ❌@626
|
|
[09:13:17] [180,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0180000.zip
|
|
[09:13:17] Reconnecting for fresh road...
|
|
[09:13:22] Connected (new road)
|
|
[09:13:37] Eval: gen_road=157.8r/1009s ❌@1009
|
|
[09:18:24] [190,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0190000.zip
|
|
[09:18:24] Reconnecting for fresh road...
|
|
[09:18:29] Connected (new road)
|
|
[09:18:55] Eval: gen_road=327.3r/2000s ✅
|
|
[09:23:45] [200,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0200000.zip
|
|
[09:23:45] Reconnecting for fresh road...
|
|
[09:23:50] Connected (new road)
|
|
[09:24:01] Eval: gen_road=107.7r/728s ❌@728
|
|
[09:28:57] [210,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0210000.zip
|
|
[09:28:57] Reconnecting for fresh road...
|
|
[09:29:02] Connected (new road)
|
|
[09:29:14] Eval: gen_road=104.2r/726s ❌@726
|
|
[09:34:15] [220,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0220000.zip
|
|
[09:34:15] Reconnecting for fresh road...
|
|
[09:34:20] Connected (new road)
|
|
[09:34:40] Eval: gen_road=207.1r/1390s ❌@1390
|
|
[09:40:21] [230,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0230000.zip
|
|
[09:40:21] Reconnecting for fresh road...
|
|
[09:40:27] Connected (new road)
|
|
[09:40:53] Eval: gen_road=290.6r/2000s ✅
|
|
[09:46:20] [240,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0240000.zip
|
|
[09:46:20] Reconnecting for fresh road...
|
|
[09:46:25] Connected (new road)
|
|
[09:46:51] Eval: gen_road=298.6r/2000s ✅
|
|
[09:52:16] [250,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0250000.zip
|
|
[09:52:16] Reconnecting for fresh road...
|
|
[09:52:21] Connected (new road)
|
|
[09:52:47] Eval: gen_road=309.2r/2000s ✅
|
|
[09:57:58] [260,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0260000.zip
|
|
[09:57:58] Reconnecting for fresh road...
|
|
[09:58:03] Connected (new road)
|
|
[09:58:30] Eval: gen_road=305.2r/2000s ✅
|
|
[10:03:51] [270,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0270000.zip
|
|
[10:03:51] Reconnecting for fresh road...
|
|
[10:03:56] Connected (new road)
|
|
[10:04:01] Eval: gen_road=22.3r/206s ❌@206
|
|
[10:09:22] [280,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0280000.zip
|
|
[10:09:22] Reconnecting for fresh road...
|
|
[10:09:27] Connected (new road)
|
|
[10:09:53] Eval: gen_road=311.9r/2000s ✅
|
|
[10:14:53] [290,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0290000.zip
|
|
[10:14:53] Reconnecting for fresh road...
|
|
[10:14:58] Connected (new road)
|
|
[10:15:24] Eval: gen_road=295.8r/2000s ✅
|
|
[10:20:23] [300,000/300,000] Checkpoint saved: /home/paulh/projects/donkeycar-rl-autoresearch/agent/models/exp26-warmstart/checkpoint_0300000.zip
|
|
[10:20:23] Reconnecting for fresh road...
|
|
[10:20:28] Connected (new road)
|
|
[10:20:42] Eval: gen_road=125.3r/879s ❌@879
|
|
[10:20:42] ============================================================
|
|
[10:20:42] FINAL EVALUATION: best_model on generated_road (3 fresh roads)
|
|
[10:20:42] ============================================================
|
|
[10:21:15] Set 1: 389.5r / 2000s ✅
|
|
[10:21:31] Set 2: 110.7r / 628s ❌@628
|
|
[10:22:03] Set 3: 389.2r / 2000s ✅
|
|
[10:22:03] Mean: 1543 steps / 296.5 reward
|
|
[10:22:03] Exp 26 complete.
|