| [2025-06-25 17:45:51,222][03575] Saving configuration to /content/train_dir/default_experiment/config.json... | |
| [2025-06-25 17:45:51,224][03575] Rollout worker 0 uses device cpu | |
| [2025-06-25 17:45:51,226][03575] Rollout worker 1 uses device cpu | |
| [2025-06-25 17:45:51,227][03575] Rollout worker 2 uses device cpu | |
| [2025-06-25 17:45:51,228][03575] Rollout worker 3 uses device cpu | |
| [2025-06-25 17:45:51,229][03575] Rollout worker 4 uses device cpu | |
| [2025-06-25 17:45:51,230][03575] Rollout worker 5 uses device cpu | |
| [2025-06-25 17:45:51,230][03575] Rollout worker 6 uses device cpu | |
| [2025-06-25 17:45:51,232][03575] Rollout worker 7 uses device cpu | |
| [2025-06-25 17:45:51,375][03575] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-06-25 17:45:51,376][03575] InferenceWorker_p0-w0: min num requests: 2 | |
| [2025-06-25 17:45:51,404][03575] Starting all processes... | |
| [2025-06-25 17:45:51,405][03575] Starting process learner_proc0 | |
| [2025-06-25 17:45:51,455][03575] Starting all processes... | |
| [2025-06-25 17:45:51,464][03575] Starting process inference_proc0-0 | |
| [2025-06-25 17:45:51,465][03575] Starting process rollout_proc0 | |
| [2025-06-25 17:45:51,469][03575] Starting process rollout_proc1 | |
| [2025-06-25 17:45:51,485][03575] Starting process rollout_proc2 | |
| [2025-06-25 17:45:51,485][03575] Starting process rollout_proc3 | |
| [2025-06-25 17:45:51,485][03575] Starting process rollout_proc4 | |
| [2025-06-25 17:45:51,486][03575] Starting process rollout_proc5 | |
| [2025-06-25 17:45:51,486][03575] Starting process rollout_proc6 | |
| [2025-06-25 17:45:51,486][03575] Starting process rollout_proc7 | |
| [2025-06-25 17:46:07,574][03782] Worker 4 uses CPU cores [0] | |
| [2025-06-25 17:46:08,027][03764] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-06-25 17:46:08,029][03764] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0 | |
| [2025-06-25 17:46:08,069][03778] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-06-25 17:46:08,072][03778] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0 | |
| [2025-06-25 17:46:08,088][03764] Num visible devices: 1 | |
| [2025-06-25 17:46:08,098][03764] Starting seed is not provided | |
| [2025-06-25 17:46:08,099][03764] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-06-25 17:46:08,099][03764] Initializing actor-critic model on device cuda:0 | |
| [2025-06-25 17:46:08,100][03764] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 17:46:08,104][03764] RunningMeanStd input shape: (1,) | |
| [2025-06-25 17:46:08,138][03778] Num visible devices: 1 | |
| [2025-06-25 17:46:08,142][03764] ConvEncoder: input_channels=3 | |
| [2025-06-25 17:46:08,402][03785] Worker 7 uses CPU cores [1] | |
| [2025-06-25 17:46:08,418][03784] Worker 6 uses CPU cores [0] | |
| [2025-06-25 17:46:08,524][03781] Worker 3 uses CPU cores [1] | |
| [2025-06-25 17:46:08,525][03783] Worker 5 uses CPU cores [1] | |
| [2025-06-25 17:46:08,549][03777] Worker 1 uses CPU cores [1] | |
| [2025-06-25 17:46:08,612][03779] Worker 0 uses CPU cores [0] | |
| [2025-06-25 17:46:08,613][03780] Worker 2 uses CPU cores [0] | |
| [2025-06-25 17:46:08,669][03764] Conv encoder output size: 512 | |
| [2025-06-25 17:46:08,669][03764] Policy head output size: 512 | |
| [2025-06-25 17:46:08,731][03764] Created Actor Critic model with architecture: | |
| [2025-06-25 17:46:08,731][03764] ActorCriticSharedWeights( | |
| (obs_normalizer): ObservationNormalizer( | |
| (running_mean_std): RunningMeanStdDictInPlace( | |
| (running_mean_std): ModuleDict( | |
| (obs): RunningMeanStdInPlace() | |
| ) | |
| ) | |
| ) | |
| (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) | |
| (encoder): VizdoomEncoder( | |
| (basic_encoder): ConvEncoder( | |
| (enc): RecursiveScriptModule( | |
| original_name=ConvEncoderImpl | |
| (conv_head): RecursiveScriptModule( | |
| original_name=Sequential | |
| (0): RecursiveScriptModule(original_name=Conv2d) | |
| (1): RecursiveScriptModule(original_name=ELU) | |
| (2): RecursiveScriptModule(original_name=Conv2d) | |
| (3): RecursiveScriptModule(original_name=ELU) | |
| (4): RecursiveScriptModule(original_name=Conv2d) | |
| (5): RecursiveScriptModule(original_name=ELU) | |
| ) | |
| (mlp_layers): RecursiveScriptModule( | |
| original_name=Sequential | |
| (0): RecursiveScriptModule(original_name=Linear) | |
| (1): RecursiveScriptModule(original_name=ELU) | |
| ) | |
| ) | |
| ) | |
| ) | |
| (core): ModelCoreRNN( | |
| (core): GRU(512, 512) | |
| ) | |
| (decoder): MlpDecoder( | |
| (mlp): Identity() | |
| ) | |
| (critic_linear): Linear(in_features=512, out_features=1, bias=True) | |
| (action_parameterization): ActionParameterizationDefault( | |
| (distribution_linear): Linear(in_features=512, out_features=5, bias=True) | |
| ) | |
| ) | |
| [2025-06-25 17:46:09,076][03764] Using optimizer <class 'torch.optim.adam.Adam'> | |
| [2025-06-25 17:46:11,375][03575] Heartbeat connected on InferenceWorker_p0-w0 | |
| [2025-06-25 17:46:11,382][03575] Heartbeat connected on RolloutWorker_w0 | |
| [2025-06-25 17:46:11,388][03575] Heartbeat connected on RolloutWorker_w2 | |
| [2025-06-25 17:46:11,390][03575] Heartbeat connected on RolloutWorker_w1 | |
| [2025-06-25 17:46:11,391][03575] Heartbeat connected on RolloutWorker_w3 | |
| [2025-06-25 17:46:11,394][03575] Heartbeat connected on RolloutWorker_w4 | |
| [2025-06-25 17:46:11,401][03575] Heartbeat connected on RolloutWorker_w6 | |
| [2025-06-25 17:46:11,402][03575] Heartbeat connected on RolloutWorker_w5 | |
| [2025-06-25 17:46:11,404][03575] Heartbeat connected on RolloutWorker_w7 | |
| [2025-06-25 17:46:11,437][03575] Heartbeat connected on Batcher_0 | |
| [2025-06-25 17:46:14,361][03764] No checkpoints found | |
| [2025-06-25 17:46:14,361][03764] Did not load from checkpoint, starting from scratch! | |
| [2025-06-25 17:46:14,362][03764] Initialized policy 0 weights for model version 0 | |
| [2025-06-25 17:46:14,364][03764] LearnerWorker_p0 finished initialization! | |
| [2025-06-25 17:46:14,365][03764] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-06-25 17:46:14,365][03575] Heartbeat connected on LearnerWorker_p0 | |
| [2025-06-25 17:46:14,593][03778] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 17:46:14,595][03778] RunningMeanStd input shape: (1,) | |
| [2025-06-25 17:46:14,608][03778] ConvEncoder: input_channels=3 | |
| [2025-06-25 17:46:14,647][03575] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-06-25 17:46:14,712][03778] Conv encoder output size: 512 | |
| [2025-06-25 17:46:14,712][03778] Policy head output size: 512 | |
| [2025-06-25 17:46:14,745][03575] Inference worker 0-0 is ready! | |
| [2025-06-25 17:46:14,746][03575] All inference workers are ready! Signal rollout workers to start! | |
| [2025-06-25 17:46:14,967][03783] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:14,985][03785] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:15,002][03777] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:15,001][03781] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:15,011][03780] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:15,013][03782] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:15,010][03784] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:15,059][03779] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 17:46:16,395][03779] Decorrelating experience for 0 frames... | |
| [2025-06-25 17:46:16,396][03784] Decorrelating experience for 0 frames... | |
| [2025-06-25 17:46:16,600][03777] Decorrelating experience for 0 frames... | |
| [2025-06-25 17:46:16,603][03781] Decorrelating experience for 0 frames... | |
| [2025-06-25 17:46:16,606][03783] Decorrelating experience for 0 frames... | |
| [2025-06-25 17:46:16,616][03785] Decorrelating experience for 0 frames... | |
| [2025-06-25 17:46:17,027][03784] Decorrelating experience for 32 frames... | |
| [2025-06-25 17:46:17,521][03783] Decorrelating experience for 32 frames... | |
| [2025-06-25 17:46:17,549][03785] Decorrelating experience for 32 frames... | |
| [2025-06-25 17:46:17,724][03782] Decorrelating experience for 0 frames... | |
| [2025-06-25 17:46:17,770][03779] Decorrelating experience for 32 frames... | |
| [2025-06-25 17:46:18,283][03781] Decorrelating experience for 32 frames... | |
| [2025-06-25 17:46:18,473][03784] Decorrelating experience for 64 frames... | |
| [2025-06-25 17:46:18,929][03782] Decorrelating experience for 32 frames... | |
| [2025-06-25 17:46:18,985][03777] Decorrelating experience for 32 frames... | |
| [2025-06-25 17:46:19,113][03783] Decorrelating experience for 64 frames... | |
| [2025-06-25 17:46:19,303][03779] Decorrelating experience for 64 frames... | |
| [2025-06-25 17:46:19,546][03784] Decorrelating experience for 96 frames... | |
| [2025-06-25 17:46:19,647][03575] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-06-25 17:46:19,990][03781] Decorrelating experience for 64 frames... | |
| [2025-06-25 17:46:20,566][03785] Decorrelating experience for 64 frames... | |
| [2025-06-25 17:46:20,661][03782] Decorrelating experience for 64 frames... | |
| [2025-06-25 17:46:20,859][03783] Decorrelating experience for 96 frames... | |
| [2025-06-25 17:46:21,068][03777] Decorrelating experience for 64 frames... | |
| [2025-06-25 17:46:21,712][03781] Decorrelating experience for 96 frames... | |
| [2025-06-25 17:46:22,395][03785] Decorrelating experience for 96 frames... | |
| [2025-06-25 17:46:22,534][03779] Decorrelating experience for 96 frames... | |
| [2025-06-25 17:46:22,589][03782] Decorrelating experience for 96 frames... | |
| [2025-06-25 17:46:24,367][03777] Decorrelating experience for 96 frames... | |
| [2025-06-25 17:46:24,647][03575] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 170.4. Samples: 1704. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-06-25 17:46:24,655][03575] Avg episode reward: [(0, '2.674')] | |
| [2025-06-25 17:46:25,219][03764] Signal inference workers to stop experience collection... | |
| [2025-06-25 17:46:25,240][03778] InferenceWorker_p0-w0: stopping experience collection | |
| [2025-06-25 17:46:27,135][03764] Signal inference workers to resume experience collection... | |
| [2025-06-25 17:46:27,135][03778] InferenceWorker_p0-w0: resuming experience collection | |
| [2025-06-25 17:46:29,647][03575] Fps is (10 sec: 1228.8, 60 sec: 819.2, 300 sec: 819.2). Total num frames: 12288. Throughput: 0: 152.7. Samples: 2290. Policy #0 lag: (min: 0.0, avg: 0.0, max: 0.0) | |
| [2025-06-25 17:46:29,649][03575] Avg episode reward: [(0, '3.075')] | |
| [2025-06-25 17:46:34,647][03575] Fps is (10 sec: 3686.4, 60 sec: 1843.2, 300 sec: 1843.2). Total num frames: 36864. Throughput: 0: 396.7. Samples: 7934. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:46:34,655][03575] Avg episode reward: [(0, '3.921')] | |
| [2025-06-25 17:46:35,191][03778] Updated weights for policy 0, policy_version 10 (0.0025) | |
| [2025-06-25 17:46:39,647][03575] Fps is (10 sec: 4505.6, 60 sec: 2293.8, 300 sec: 2293.8). Total num frames: 57344. Throughput: 0: 587.8. Samples: 14694. Policy #0 lag: (min: 0.0, avg: 0.7, max: 1.0) | |
| [2025-06-25 17:46:39,649][03575] Avg episode reward: [(0, '4.466')] | |
| [2025-06-25 17:46:44,647][03575] Fps is (10 sec: 3686.4, 60 sec: 2457.6, 300 sec: 2457.6). Total num frames: 73728. Throughput: 0: 537.3. Samples: 16118. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:46:44,651][03575] Avg episode reward: [(0, '4.416')] | |
| [2025-06-25 17:46:46,361][03778] Updated weights for policy 0, policy_version 20 (0.0013) | |
| [2025-06-25 17:46:49,647][03575] Fps is (10 sec: 3686.4, 60 sec: 2691.7, 300 sec: 2691.7). Total num frames: 94208. Throughput: 0: 644.6. Samples: 22562. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:46:49,652][03575] Avg episode reward: [(0, '4.350')] | |
| [2025-06-25 17:46:54,654][03575] Fps is (10 sec: 4093.1, 60 sec: 2866.7, 300 sec: 2866.7). Total num frames: 114688. Throughput: 0: 728.9. Samples: 29160. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:46:54,656][03575] Avg episode reward: [(0, '4.307')] | |
| [2025-06-25 17:46:54,667][03764] Saving new best policy, reward=4.307! | |
| [2025-06-25 17:46:56,914][03778] Updated weights for policy 0, policy_version 30 (0.0021) | |
| [2025-06-25 17:46:59,647][03575] Fps is (10 sec: 3686.4, 60 sec: 2912.7, 300 sec: 2912.7). Total num frames: 131072. Throughput: 0: 690.2. Samples: 31060. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:46:59,649][03575] Avg episode reward: [(0, '4.371')] | |
| [2025-06-25 17:46:59,652][03764] Saving new best policy, reward=4.371! | |
| [2025-06-25 17:47:04,647][03575] Fps is (10 sec: 3689.0, 60 sec: 3031.0, 300 sec: 3031.0). Total num frames: 151552. Throughput: 0: 828.4. Samples: 37276. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:47:04,651][03575] Avg episode reward: [(0, '4.384')] | |
| [2025-06-25 17:47:04,667][03764] Saving new best policy, reward=4.384! | |
| [2025-06-25 17:47:06,289][03778] Updated weights for policy 0, policy_version 40 (0.0023) | |
| [2025-06-25 17:47:09,654][03575] Fps is (10 sec: 4093.2, 60 sec: 3127.5, 300 sec: 3127.5). Total num frames: 172032. Throughput: 0: 934.8. Samples: 43776. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:47:09,656][03575] Avg episode reward: [(0, '4.414')] | |
| [2025-06-25 17:47:09,664][03764] Saving new best policy, reward=4.414! | |
| [2025-06-25 17:47:14,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3208.5, 300 sec: 3208.5). Total num frames: 192512. Throughput: 0: 966.6. Samples: 45786. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:47:14,648][03575] Avg episode reward: [(0, '4.403')] | |
| [2025-06-25 17:47:17,008][03778] Updated weights for policy 0, policy_version 50 (0.0019) | |
| [2025-06-25 17:47:19,647][03575] Fps is (10 sec: 4508.7, 60 sec: 3618.1, 300 sec: 3339.8). Total num frames: 217088. Throughput: 0: 994.7. Samples: 52696. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:47:19,650][03575] Avg episode reward: [(0, '4.431')] | |
| [2025-06-25 17:47:19,653][03764] Saving new best policy, reward=4.431! | |
| [2025-06-25 17:47:24,651][03575] Fps is (10 sec: 4094.4, 60 sec: 3891.0, 300 sec: 3335.1). Total num frames: 233472. Throughput: 0: 979.0. Samples: 58752. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:47:24,653][03575] Avg episode reward: [(0, '4.416')] | |
| [2025-06-25 17:47:27,843][03778] Updated weights for policy 0, policy_version 60 (0.0032) | |
| [2025-06-25 17:47:29,648][03575] Fps is (10 sec: 3276.7, 60 sec: 3959.4, 300 sec: 3331.4). Total num frames: 249856. Throughput: 0: 990.7. Samples: 60700. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:47:29,652][03575] Avg episode reward: [(0, '4.293')] | |
| [2025-06-25 17:47:34,647][03575] Fps is (10 sec: 4097.6, 60 sec: 3959.5, 300 sec: 3430.4). Total num frames: 274432. Throughput: 0: 993.4. Samples: 67266. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:47:34,649][03575] Avg episode reward: [(0, '4.404')] | |
| [2025-06-25 17:47:37,158][03778] Updated weights for policy 0, policy_version 70 (0.0025) | |
| [2025-06-25 17:47:39,647][03575] Fps is (10 sec: 4096.1, 60 sec: 3891.2, 300 sec: 3421.4). Total num frames: 290816. Throughput: 0: 976.4. Samples: 73090. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:47:39,651][03575] Avg episode reward: [(0, '4.487')] | |
| [2025-06-25 17:47:39,654][03764] Saving new best policy, reward=4.487! | |
| [2025-06-25 17:47:44,647][03575] Fps is (10 sec: 3276.8, 60 sec: 3891.2, 300 sec: 3413.3). Total num frames: 307200. Throughput: 0: 962.1. Samples: 74356. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:47:44,648][03575] Avg episode reward: [(0, '4.494')] | |
| [2025-06-25 17:47:44,657][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000075_307200.pth... | |
| [2025-06-25 17:47:44,814][03764] Saving new best policy, reward=4.494! | |
| [2025-06-25 17:47:49,111][03778] Updated weights for policy 0, policy_version 80 (0.0053) | |
| [2025-06-25 17:47:49,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3449.3). Total num frames: 327680. Throughput: 0: 965.7. Samples: 80734. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:47:49,649][03575] Avg episode reward: [(0, '4.692')] | |
| [2025-06-25 17:47:49,653][03764] Saving new best policy, reward=4.692! | |
| [2025-06-25 17:47:54,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3823.4, 300 sec: 3440.6). Total num frames: 344064. Throughput: 0: 945.2. Samples: 86302. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:47:54,651][03575] Avg episode reward: [(0, '4.596')] | |
| [2025-06-25 17:47:59,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3471.9). Total num frames: 364544. Throughput: 0: 950.7. Samples: 88566. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:47:59,649][03575] Avg episode reward: [(0, '4.562')] | |
| [2025-06-25 17:48:00,371][03778] Updated weights for policy 0, policy_version 90 (0.0022) | |
| [2025-06-25 17:48:04,648][03575] Fps is (10 sec: 4095.9, 60 sec: 3891.2, 300 sec: 3500.2). Total num frames: 385024. Throughput: 0: 946.6. Samples: 95292. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:48:04,649][03575] Avg episode reward: [(0, '4.640')] | |
| [2025-06-25 17:48:09,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3823.4, 300 sec: 3490.5). Total num frames: 401408. Throughput: 0: 933.9. Samples: 100776. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) | |
| [2025-06-25 17:48:09,650][03575] Avg episode reward: [(0, '4.606')] | |
| [2025-06-25 17:48:11,158][03778] Updated weights for policy 0, policy_version 100 (0.0012) | |
| [2025-06-25 17:48:14,647][03575] Fps is (10 sec: 3686.5, 60 sec: 3822.9, 300 sec: 3515.7). Total num frames: 421888. Throughput: 0: 952.6. Samples: 103568. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:48:14,651][03575] Avg episode reward: [(0, '4.693')] | |
| [2025-06-25 17:48:14,714][03764] Saving new best policy, reward=4.693! | |
| [2025-06-25 17:48:19,647][03575] Fps is (10 sec: 4505.6, 60 sec: 3822.9, 300 sec: 3571.7). Total num frames: 446464. Throughput: 0: 962.6. Samples: 110582. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:48:19,650][03575] Avg episode reward: [(0, '4.822')] | |
| [2025-06-25 17:48:19,658][03764] Saving new best policy, reward=4.822! | |
| [2025-06-25 17:48:19,935][03778] Updated weights for policy 0, policy_version 110 (0.0021) | |
| [2025-06-25 17:48:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3823.2, 300 sec: 3560.4). Total num frames: 462848. Throughput: 0: 945.8. Samples: 115652. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:48:24,649][03575] Avg episode reward: [(0, '4.900')] | |
| [2025-06-25 17:48:24,656][03764] Saving new best policy, reward=4.900! | |
| [2025-06-25 17:48:29,647][03575] Fps is (10 sec: 3686.3, 60 sec: 3891.2, 300 sec: 3580.2). Total num frames: 483328. Throughput: 0: 983.5. Samples: 118614. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:48:29,649][03575] Avg episode reward: [(0, '4.790')] | |
| [2025-06-25 17:48:30,789][03778] Updated weights for policy 0, policy_version 120 (0.0027) | |
| [2025-06-25 17:48:34,650][03575] Fps is (10 sec: 4504.3, 60 sec: 3891.0, 300 sec: 3627.8). Total num frames: 507904. Throughput: 0: 997.9. Samples: 125642. Policy #0 lag: (min: 0.0, avg: 0.2, max: 1.0) | |
| [2025-06-25 17:48:34,657][03575] Avg episode reward: [(0, '4.758')] | |
| [2025-06-25 17:48:39,647][03575] Fps is (10 sec: 4096.1, 60 sec: 3891.2, 300 sec: 3615.8). Total num frames: 524288. Throughput: 0: 985.1. Samples: 130632. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:48:39,652][03575] Avg episode reward: [(0, '5.001')] | |
| [2025-06-25 17:48:39,655][03764] Saving new best policy, reward=5.001! | |
| [2025-06-25 17:48:41,490][03778] Updated weights for policy 0, policy_version 130 (0.0012) | |
| [2025-06-25 17:48:44,647][03575] Fps is (10 sec: 3687.4, 60 sec: 3959.5, 300 sec: 3631.8). Total num frames: 544768. Throughput: 0: 1008.9. Samples: 133968. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:48:44,649][03575] Avg episode reward: [(0, '4.787')] | |
| [2025-06-25 17:48:49,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4027.7, 300 sec: 3673.2). Total num frames: 569344. Throughput: 0: 1014.5. Samples: 140946. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:48:49,649][03575] Avg episode reward: [(0, '4.608')] | |
| [2025-06-25 17:48:50,922][03778] Updated weights for policy 0, policy_version 140 (0.0024) | |
| [2025-06-25 17:48:54,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 3660.8). Total num frames: 585728. Throughput: 0: 1006.0. Samples: 146044. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:48:54,653][03575] Avg episode reward: [(0, '4.685')] | |
| [2025-06-25 17:48:59,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.7, 300 sec: 3674.0). Total num frames: 606208. Throughput: 0: 1022.0. Samples: 149560. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:48:59,650][03575] Avg episode reward: [(0, '4.623')] | |
| [2025-06-25 17:49:00,673][03778] Updated weights for policy 0, policy_version 150 (0.0014) | |
| [2025-06-25 17:49:04,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 3686.4). Total num frames: 626688. Throughput: 0: 1017.3. Samples: 156360. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:49:04,649][03575] Avg episode reward: [(0, '4.637')] | |
| [2025-06-25 17:49:09,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 3698.1). Total num frames: 647168. Throughput: 0: 1015.2. Samples: 161338. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:49:09,650][03575] Avg episode reward: [(0, '4.648')] | |
| [2025-06-25 17:49:11,231][03778] Updated weights for policy 0, policy_version 160 (0.0018) | |
| [2025-06-25 17:49:14,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 3709.2). Total num frames: 667648. Throughput: 0: 1026.4. Samples: 164800. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:49:14,648][03575] Avg episode reward: [(0, '4.793')] | |
| [2025-06-25 17:49:19,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 3719.6). Total num frames: 688128. Throughput: 0: 1028.0. Samples: 171898. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:49:19,649][03575] Avg episode reward: [(0, '4.695')] | |
| [2025-06-25 17:49:21,168][03778] Updated weights for policy 0, policy_version 170 (0.0027) | |
| [2025-06-25 17:49:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 3729.5). Total num frames: 708608. Throughput: 0: 1029.8. Samples: 176972. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:49:24,649][03575] Avg episode reward: [(0, '4.523')] | |
| [2025-06-25 17:49:29,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 3738.9). Total num frames: 729088. Throughput: 0: 1033.7. Samples: 180486. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:49:29,649][03575] Avg episode reward: [(0, '4.642')] | |
| [2025-06-25 17:49:30,683][03778] Updated weights for policy 0, policy_version 180 (0.0014) | |
| [2025-06-25 17:49:34,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.9, 300 sec: 3747.8). Total num frames: 749568. Throughput: 0: 1025.2. Samples: 187080. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:49:34,649][03575] Avg episode reward: [(0, '4.878')] | |
| [2025-06-25 17:49:39,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 3756.3). Total num frames: 770048. Throughput: 0: 1030.5. Samples: 192416. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:49:39,649][03575] Avg episode reward: [(0, '4.634')] | |
| [2025-06-25 17:49:41,026][03778] Updated weights for policy 0, policy_version 190 (0.0016) | |
| [2025-06-25 17:49:44,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4164.3, 300 sec: 3783.9). Total num frames: 794624. Throughput: 0: 1029.8. Samples: 195900. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:49:44,653][03575] Avg episode reward: [(0, '4.400')] | |
| [2025-06-25 17:49:44,663][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000194_794624.pth... | |
| [2025-06-25 17:49:49,648][03575] Fps is (10 sec: 4095.9, 60 sec: 4027.7, 300 sec: 3772.1). Total num frames: 811008. Throughput: 0: 1023.1. Samples: 202398. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:49:49,653][03575] Avg episode reward: [(0, '4.714')] | |
| [2025-06-25 17:49:51,511][03778] Updated weights for policy 0, policy_version 200 (0.0012) | |
| [2025-06-25 17:49:54,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 3779.5). Total num frames: 831488. Throughput: 0: 1037.3. Samples: 208016. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:49:54,651][03575] Avg episode reward: [(0, '4.898')] | |
| [2025-06-25 17:49:59,647][03575] Fps is (10 sec: 4505.7, 60 sec: 4164.3, 300 sec: 3804.7). Total num frames: 856064. Throughput: 0: 1038.4. Samples: 211530. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:49:59,649][03575] Avg episode reward: [(0, '4.717')] | |
| [2025-06-25 17:50:00,326][03778] Updated weights for policy 0, policy_version 210 (0.0017) | |
| [2025-06-25 17:50:04,647][03575] Fps is (10 sec: 4095.9, 60 sec: 4096.0, 300 sec: 3793.3). Total num frames: 872448. Throughput: 0: 1015.6. Samples: 217598. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:50:04,651][03575] Avg episode reward: [(0, '4.726')] | |
| [2025-06-25 17:50:09,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 3799.7). Total num frames: 892928. Throughput: 0: 1031.2. Samples: 223376. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:50:09,649][03575] Avg episode reward: [(0, '4.499')] | |
| [2025-06-25 17:50:11,144][03778] Updated weights for policy 0, policy_version 220 (0.0016) | |
| [2025-06-25 17:50:14,647][03575] Fps is (10 sec: 4096.1, 60 sec: 4096.0, 300 sec: 3805.9). Total num frames: 913408. Throughput: 0: 1023.7. Samples: 226554. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) | |
| [2025-06-25 17:50:14,649][03575] Avg episode reward: [(0, '4.763')] | |
| [2025-06-25 17:50:19,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.7, 300 sec: 3795.1). Total num frames: 929792. Throughput: 0: 1007.1. Samples: 232400. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:50:19,649][03575] Avg episode reward: [(0, '4.839')] | |
| [2025-06-25 17:50:21,640][03778] Updated weights for policy 0, policy_version 230 (0.0041) | |
| [2025-06-25 17:50:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 3817.5). Total num frames: 954368. Throughput: 0: 1022.7. Samples: 238436. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:50:24,649][03575] Avg episode reward: [(0, '4.891')] | |
| [2025-06-25 17:50:29,648][03575] Fps is (10 sec: 4914.7, 60 sec: 4164.2, 300 sec: 3839.0). Total num frames: 978944. Throughput: 0: 1024.7. Samples: 242014. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:50:29,651][03575] Avg episode reward: [(0, '4.953')] | |
| [2025-06-25 17:50:30,636][03778] Updated weights for policy 0, policy_version 240 (0.0019) | |
| [2025-06-25 17:50:34,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.7, 300 sec: 3812.4). Total num frames: 991232. Throughput: 0: 1004.2. Samples: 247588. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:50:34,649][03575] Avg episode reward: [(0, '4.799')] | |
| [2025-06-25 17:50:39,647][03575] Fps is (10 sec: 3686.7, 60 sec: 4096.0, 300 sec: 3833.2). Total num frames: 1015808. Throughput: 0: 1020.4. Samples: 253934. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:50:39,649][03575] Avg episode reward: [(0, '5.060')] | |
| [2025-06-25 17:50:39,653][03764] Saving new best policy, reward=5.060! | |
| [2025-06-25 17:50:40,924][03778] Updated weights for policy 0, policy_version 250 (0.0019) | |
| [2025-06-25 17:50:44,649][03575] Fps is (10 sec: 4504.7, 60 sec: 4027.6, 300 sec: 3838.1). Total num frames: 1036288. Throughput: 0: 1018.1. Samples: 257348. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) | |
| [2025-06-25 17:50:44,650][03575] Avg episode reward: [(0, '5.502')] | |
| [2025-06-25 17:50:44,662][03764] Saving new best policy, reward=5.502! | |
| [2025-06-25 17:50:49,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.7, 300 sec: 3827.9). Total num frames: 1052672. Throughput: 0: 997.0. Samples: 262462. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:50:49,649][03575] Avg episode reward: [(0, '5.475')] | |
| [2025-06-25 17:50:51,978][03778] Updated weights for policy 0, policy_version 260 (0.0017) | |
| [2025-06-25 17:50:54,647][03575] Fps is (10 sec: 3687.1, 60 sec: 4027.7, 300 sec: 3832.7). Total num frames: 1073152. Throughput: 0: 1007.5. Samples: 268714. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:50:54,652][03575] Avg episode reward: [(0, '5.226')] | |
| [2025-06-25 17:50:59,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3959.5, 300 sec: 3837.3). Total num frames: 1093632. Throughput: 0: 1012.2. Samples: 272102. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:50:59,650][03575] Avg episode reward: [(0, '5.559')] | |
| [2025-06-25 17:50:59,653][03764] Saving new best policy, reward=5.559! | |
| [2025-06-25 17:51:02,971][03778] Updated weights for policy 0, policy_version 270 (0.0018) | |
| [2025-06-25 17:51:04,648][03575] Fps is (10 sec: 3686.3, 60 sec: 3959.5, 300 sec: 3827.6). Total num frames: 1110016. Throughput: 0: 987.4. Samples: 276834. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:51:04,649][03575] Avg episode reward: [(0, '5.730')] | |
| [2025-06-25 17:51:04,656][03764] Saving new best policy, reward=5.730! | |
| [2025-06-25 17:51:09,648][03575] Fps is (10 sec: 4095.9, 60 sec: 4027.7, 300 sec: 3846.1). Total num frames: 1134592. Throughput: 0: 998.6. Samples: 283372. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:51:09,657][03575] Avg episode reward: [(0, '5.414')] | |
| [2025-06-25 17:51:12,267][03778] Updated weights for policy 0, policy_version 280 (0.0015) | |
| [2025-06-25 17:51:14,649][03575] Fps is (10 sec: 4504.9, 60 sec: 4027.6, 300 sec: 3915.5). Total num frames: 1155072. Throughput: 0: 992.9. Samples: 286694. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:51:14,651][03575] Avg episode reward: [(0, '5.167')] | |
| [2025-06-25 17:51:19,647][03575] Fps is (10 sec: 3686.5, 60 sec: 4027.7, 300 sec: 3971.0). Total num frames: 1171456. Throughput: 0: 978.3. Samples: 291612. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:51:19,649][03575] Avg episode reward: [(0, '5.341')] | |
| [2025-06-25 17:51:23,219][03778] Updated weights for policy 0, policy_version 290 (0.0033) | |
| [2025-06-25 17:51:24,647][03575] Fps is (10 sec: 3687.1, 60 sec: 3959.5, 300 sec: 3998.8). Total num frames: 1191936. Throughput: 0: 982.0. Samples: 298122. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:51:24,649][03575] Avg episode reward: [(0, '4.970')] | |
| [2025-06-25 17:51:29,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3891.3, 300 sec: 3984.9). Total num frames: 1212416. Throughput: 0: 978.4. Samples: 301372. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:51:29,649][03575] Avg episode reward: [(0, '5.227')] | |
| [2025-06-25 17:51:34,479][03778] Updated weights for policy 0, policy_version 300 (0.0016) | |
| [2025-06-25 17:51:34,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3959.5, 300 sec: 3971.0). Total num frames: 1228800. Throughput: 0: 964.4. Samples: 305862. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:51:34,651][03575] Avg episode reward: [(0, '5.266')] | |
| [2025-06-25 17:51:39,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3984.9). Total num frames: 1249280. Throughput: 0: 974.8. Samples: 312578. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:51:39,649][03575] Avg episode reward: [(0, '5.293')] | |
| [2025-06-25 17:51:44,429][03778] Updated weights for policy 0, policy_version 310 (0.0016) | |
| [2025-06-25 17:51:44,648][03575] Fps is (10 sec: 4095.9, 60 sec: 3891.3, 300 sec: 3984.9). Total num frames: 1269760. Throughput: 0: 974.6. Samples: 315960. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:51:44,650][03575] Avg episode reward: [(0, '5.541')] | |
| [2025-06-25 17:51:44,658][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000310_1269760.pth... | |
| [2025-06-25 17:51:44,823][03764] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000075_307200.pth | |
| [2025-06-25 17:51:49,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3971.1). Total num frames: 1286144. Throughput: 0: 977.3. Samples: 320812. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:51:49,649][03575] Avg episode reward: [(0, '5.583')] | |
| [2025-06-25 17:51:54,548][03778] Updated weights for policy 0, policy_version 320 (0.0021) | |
| [2025-06-25 17:51:54,647][03575] Fps is (10 sec: 4096.1, 60 sec: 3959.5, 300 sec: 3998.8). Total num frames: 1310720. Throughput: 0: 977.7. Samples: 327370. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:51:54,649][03575] Avg episode reward: [(0, '5.406')] | |
| [2025-06-25 17:51:59,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3984.9). Total num frames: 1327104. Throughput: 0: 976.7. Samples: 330642. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:51:59,650][03575] Avg episode reward: [(0, '5.366')] | |
| [2025-06-25 17:52:04,647][03575] Fps is (10 sec: 3276.8, 60 sec: 3891.2, 300 sec: 3971.1). Total num frames: 1343488. Throughput: 0: 965.7. Samples: 335070. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:52:04,649][03575] Avg episode reward: [(0, '5.310')] | |
| [2025-06-25 17:52:05,868][03778] Updated weights for policy 0, policy_version 330 (0.0023) | |
| [2025-06-25 17:52:09,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3984.9). Total num frames: 1368064. Throughput: 0: 965.6. Samples: 341572. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:52:09,652][03575] Avg episode reward: [(0, '5.513')] | |
| [2025-06-25 17:52:14,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3823.1, 300 sec: 3957.2). Total num frames: 1384448. Throughput: 0: 965.4. Samples: 344814. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:52:14,651][03575] Avg episode reward: [(0, '5.747')] | |
| [2025-06-25 17:52:14,660][03764] Saving new best policy, reward=5.747! | |
| [2025-06-25 17:52:16,744][03778] Updated weights for policy 0, policy_version 340 (0.0019) | |
| [2025-06-25 17:52:19,650][03575] Fps is (10 sec: 3685.4, 60 sec: 3891.0, 300 sec: 3971.1). Total num frames: 1404928. Throughput: 0: 975.2. Samples: 349748. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:52:19,652][03575] Avg episode reward: [(0, '5.668')] | |
| [2025-06-25 17:52:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3984.9). Total num frames: 1425408. Throughput: 0: 976.3. Samples: 356510. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:52:24,649][03575] Avg episode reward: [(0, '5.107')] | |
| [2025-06-25 17:52:25,845][03778] Updated weights for policy 0, policy_version 350 (0.0012) | |
| [2025-06-25 17:52:29,647][03575] Fps is (10 sec: 3687.4, 60 sec: 3822.9, 300 sec: 3957.2). Total num frames: 1441792. Throughput: 0: 970.9. Samples: 359652. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:52:29,649][03575] Avg episode reward: [(0, '5.539')] | |
| [2025-06-25 17:52:34,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3891.2, 300 sec: 3971.0). Total num frames: 1462272. Throughput: 0: 970.9. Samples: 364504. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:52:34,649][03575] Avg episode reward: [(0, '5.845')] | |
| [2025-06-25 17:52:34,654][03764] Saving new best policy, reward=5.845! | |
| [2025-06-25 17:52:36,972][03778] Updated weights for policy 0, policy_version 360 (0.0023) | |
| [2025-06-25 17:52:39,648][03575] Fps is (10 sec: 4505.5, 60 sec: 3959.5, 300 sec: 3998.8). Total num frames: 1486848. Throughput: 0: 973.9. Samples: 371196. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:52:39,649][03575] Avg episode reward: [(0, '5.824')] | |
| [2025-06-25 17:52:44,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3891.2, 300 sec: 3984.9). Total num frames: 1503232. Throughput: 0: 970.3. Samples: 374306. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:52:44,651][03575] Avg episode reward: [(0, '5.729')] | |
| [2025-06-25 17:52:47,502][03778] Updated weights for policy 0, policy_version 370 (0.0023) | |
| [2025-06-25 17:52:49,647][03575] Fps is (10 sec: 3686.5, 60 sec: 3959.5, 300 sec: 3998.8). Total num frames: 1523712. Throughput: 0: 994.3. Samples: 379812. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:52:49,652][03575] Avg episode reward: [(0, '5.936')] | |
| [2025-06-25 17:52:49,656][03764] Saving new best policy, reward=5.936! | |
| [2025-06-25 17:52:54,647][03575] Fps is (10 sec: 4505.6, 60 sec: 3959.5, 300 sec: 4012.7). Total num frames: 1548288. Throughput: 0: 1006.6. Samples: 386868. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:52:54,652][03575] Avg episode reward: [(0, '6.475')] | |
| [2025-06-25 17:52:54,663][03764] Saving new best policy, reward=6.475! | |
| [2025-06-25 17:52:56,715][03778] Updated weights for policy 0, policy_version 380 (0.0019) | |
| [2025-06-25 17:52:59,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3959.5, 300 sec: 3998.8). Total num frames: 1564672. Throughput: 0: 995.1. Samples: 389592. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:52:59,652][03575] Avg episode reward: [(0, '6.573')] | |
| [2025-06-25 17:52:59,656][03764] Saving new best policy, reward=6.573! | |
| [2025-06-25 17:53:04,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.7, 300 sec: 4012.7). Total num frames: 1585152. Throughput: 0: 1011.2. Samples: 395250. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:53:04,649][03575] Avg episode reward: [(0, '6.301')] | |
| [2025-06-25 17:53:06,750][03778] Updated weights for policy 0, policy_version 390 (0.0019) | |
| [2025-06-25 17:53:09,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4027.7, 300 sec: 4026.6). Total num frames: 1609728. Throughput: 0: 1018.5. Samples: 402342. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:53:09,650][03575] Avg episode reward: [(0, '6.680')] | |
| [2025-06-25 17:53:09,657][03764] Saving new best policy, reward=6.680! | |
| [2025-06-25 17:53:14,648][03575] Fps is (10 sec: 4095.9, 60 sec: 4027.7, 300 sec: 3998.8). Total num frames: 1626112. Throughput: 0: 1004.8. Samples: 404866. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:53:14,652][03575] Avg episode reward: [(0, '7.311')] | |
| [2025-06-25 17:53:14,659][03764] Saving new best policy, reward=7.311! | |
| [2025-06-25 17:53:17,175][03778] Updated weights for policy 0, policy_version 400 (0.0028) | |
| [2025-06-25 17:53:19,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.2, 300 sec: 4026.6). Total num frames: 1650688. Throughput: 0: 1031.1. Samples: 410902. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:53:19,649][03575] Avg episode reward: [(0, '6.933')] | |
| [2025-06-25 17:53:24,647][03575] Fps is (10 sec: 4505.7, 60 sec: 4096.0, 300 sec: 4026.6). Total num frames: 1671168. Throughput: 0: 1040.6. Samples: 418024. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:53:24,648][03575] Avg episode reward: [(0, '6.951')] | |
| [2025-06-25 17:53:26,990][03778] Updated weights for policy 0, policy_version 410 (0.0025) | |
| [2025-06-25 17:53:29,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 3998.8). Total num frames: 1687552. Throughput: 0: 1021.3. Samples: 420264. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:53:29,649][03575] Avg episode reward: [(0, '6.806')] | |
| [2025-06-25 17:53:34,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4026.6). Total num frames: 1712128. Throughput: 0: 1037.6. Samples: 426506. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:53:34,652][03575] Avg episode reward: [(0, '6.436')] | |
| [2025-06-25 17:53:35,985][03778] Updated weights for policy 0, policy_version 420 (0.0025) | |
| [2025-06-25 17:53:39,648][03575] Fps is (10 sec: 4505.1, 60 sec: 4095.9, 300 sec: 4026.6). Total num frames: 1732608. Throughput: 0: 1037.0. Samples: 433532. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:53:39,650][03575] Avg episode reward: [(0, '6.641')] | |
| [2025-06-25 17:53:44,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 3998.8). Total num frames: 1748992. Throughput: 0: 1023.0. Samples: 435628. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:53:44,649][03575] Avg episode reward: [(0, '6.898')] | |
| [2025-06-25 17:53:44,658][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000427_1748992.pth... | |
| [2025-06-25 17:53:44,776][03764] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000194_794624.pth | |
| [2025-06-25 17:53:46,647][03778] Updated weights for policy 0, policy_version 430 (0.0014) | |
| [2025-06-25 17:53:49,647][03575] Fps is (10 sec: 4096.4, 60 sec: 4164.3, 300 sec: 4026.6). Total num frames: 1773568. Throughput: 0: 1043.0. Samples: 442184. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:53:49,649][03575] Avg episode reward: [(0, '7.540')] | |
| [2025-06-25 17:53:49,650][03764] Saving new best policy, reward=7.540! | |
| [2025-06-25 17:53:54,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4026.6). Total num frames: 1794048. Throughput: 0: 1031.1. Samples: 448740. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:53:54,649][03575] Avg episode reward: [(0, '7.285')] | |
| [2025-06-25 17:53:57,215][03778] Updated weights for policy 0, policy_version 440 (0.0022) | |
| [2025-06-25 17:53:59,648][03575] Fps is (10 sec: 3686.1, 60 sec: 4095.9, 300 sec: 4012.7). Total num frames: 1810432. Throughput: 0: 1021.6. Samples: 450838. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:53:59,649][03575] Avg episode reward: [(0, '7.480')] | |
| [2025-06-25 17:54:04,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4012.7). Total num frames: 1830912. Throughput: 0: 1028.0. Samples: 457160. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:54:04,649][03575] Avg episode reward: [(0, '8.248')] | |
| [2025-06-25 17:54:04,684][03764] Saving new best policy, reward=8.248! | |
| [2025-06-25 17:54:06,478][03778] Updated weights for policy 0, policy_version 450 (0.0022) | |
| [2025-06-25 17:54:09,648][03575] Fps is (10 sec: 4096.2, 60 sec: 4027.7, 300 sec: 4012.7). Total num frames: 1851392. Throughput: 0: 1005.2. Samples: 463256. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:54:09,652][03575] Avg episode reward: [(0, '8.802')] | |
| [2025-06-25 17:54:09,654][03764] Saving new best policy, reward=8.802! | |
| [2025-06-25 17:54:14,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 4012.7). Total num frames: 1871872. Throughput: 0: 1002.8. Samples: 465388. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:54:14,651][03575] Avg episode reward: [(0, '9.269')] | |
| [2025-06-25 17:54:14,658][03764] Saving new best policy, reward=9.269! | |
| [2025-06-25 17:54:17,253][03778] Updated weights for policy 0, policy_version 460 (0.0014) | |
| [2025-06-25 17:54:19,648][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 4012.7). Total num frames: 1892352. Throughput: 0: 1015.4. Samples: 472198. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:54:19,649][03575] Avg episode reward: [(0, '8.717')] | |
| [2025-06-25 17:54:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 4012.7). Total num frames: 1912832. Throughput: 0: 995.3. Samples: 478320. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:54:24,649][03575] Avg episode reward: [(0, '8.900')] | |
| [2025-06-25 17:54:27,541][03778] Updated weights for policy 0, policy_version 470 (0.0021) | |
| [2025-06-25 17:54:29,647][03575] Fps is (10 sec: 4096.1, 60 sec: 4096.0, 300 sec: 4012.7). Total num frames: 1933312. Throughput: 0: 1002.1. Samples: 480724. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:54:29,649][03575] Avg episode reward: [(0, '9.327')] | |
| [2025-06-25 17:54:29,653][03764] Saving new best policy, reward=9.327! | |
| [2025-06-25 17:54:34,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 4012.7). Total num frames: 1953792. Throughput: 0: 1005.4. Samples: 487426. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:54:34,655][03575] Avg episode reward: [(0, '9.899')] | |
| [2025-06-25 17:54:34,662][03764] Saving new best policy, reward=9.899! | |
| [2025-06-25 17:54:37,355][03778] Updated weights for policy 0, policy_version 480 (0.0022) | |
| [2025-06-25 17:54:39,647][03575] Fps is (10 sec: 3686.4, 60 sec: 3959.5, 300 sec: 3984.9). Total num frames: 1970176. Throughput: 0: 984.7. Samples: 493050. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:54:39,651][03575] Avg episode reward: [(0, '10.421')] | |
| [2025-06-25 17:54:39,655][03764] Saving new best policy, reward=10.421! | |
| [2025-06-25 17:54:44,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.7, 300 sec: 3998.8). Total num frames: 1990656. Throughput: 0: 995.0. Samples: 495610. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:54:44,653][03575] Avg episode reward: [(0, '11.005')] | |
| [2025-06-25 17:54:44,670][03764] Saving new best policy, reward=11.005! | |
| [2025-06-25 17:54:47,519][03778] Updated weights for policy 0, policy_version 490 (0.0017) | |
| [2025-06-25 17:54:49,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4027.7, 300 sec: 4012.7). Total num frames: 2015232. Throughput: 0: 1009.3. Samples: 502578. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) | |
| [2025-06-25 17:54:49,649][03575] Avg episode reward: [(0, '10.906')] | |
| [2025-06-25 17:54:54,647][03575] Fps is (10 sec: 4096.0, 60 sec: 3959.5, 300 sec: 3984.9). Total num frames: 2031616. Throughput: 0: 997.7. Samples: 508152. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:54:54,652][03575] Avg episode reward: [(0, '10.871')] | |
| [2025-06-25 17:54:57,910][03778] Updated weights for policy 0, policy_version 500 (0.0028) | |
| [2025-06-25 17:54:59,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.8, 300 sec: 3998.8). Total num frames: 2052096. Throughput: 0: 1015.6. Samples: 511092. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:54:59,649][03575] Avg episode reward: [(0, '10.591')] | |
| [2025-06-25 17:55:04,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4012.7). Total num frames: 2076672. Throughput: 0: 1018.4. Samples: 518028. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:55:04,650][03575] Avg episode reward: [(0, '11.157')] | |
| [2025-06-25 17:55:04,663][03764] Saving new best policy, reward=11.157! | |
| [2025-06-25 17:55:07,752][03778] Updated weights for policy 0, policy_version 510 (0.0014) | |
| [2025-06-25 17:55:09,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 3998.8). Total num frames: 2093056. Throughput: 0: 1001.8. Samples: 523402. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:55:09,649][03575] Avg episode reward: [(0, '11.111')] | |
| [2025-06-25 17:55:14,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4027.7, 300 sec: 4012.7). Total num frames: 2113536. Throughput: 0: 1018.7. Samples: 526564. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:55:14,652][03575] Avg episode reward: [(0, '11.170')] | |
| [2025-06-25 17:55:14,725][03764] Saving new best policy, reward=11.170! | |
| [2025-06-25 17:55:17,045][03778] Updated weights for policy 0, policy_version 520 (0.0022) | |
| [2025-06-25 17:55:19,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4012.7). Total num frames: 2138112. Throughput: 0: 1027.3. Samples: 533656. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:55:19,649][03575] Avg episode reward: [(0, '11.602')] | |
| [2025-06-25 17:55:19,652][03764] Saving new best policy, reward=11.602! | |
| [2025-06-25 17:55:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4027.7, 300 sec: 3984.9). Total num frames: 2154496. Throughput: 0: 1016.8. Samples: 538806. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:55:24,648][03575] Avg episode reward: [(0, '11.431')] | |
| [2025-06-25 17:55:27,534][03778] Updated weights for policy 0, policy_version 530 (0.0019) | |
| [2025-06-25 17:55:29,647][03575] Fps is (10 sec: 4095.9, 60 sec: 4096.0, 300 sec: 4026.6). Total num frames: 2179072. Throughput: 0: 1036.4. Samples: 542246. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:55:29,649][03575] Avg episode reward: [(0, '12.448')] | |
| [2025-06-25 17:55:29,653][03764] Saving new best policy, reward=12.448! | |
| [2025-06-25 17:55:34,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4012.7). Total num frames: 2199552. Throughput: 0: 1036.8. Samples: 549232. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:55:34,649][03575] Avg episode reward: [(0, '13.281')] | |
| [2025-06-25 17:55:34,656][03764] Saving new best policy, reward=13.281! | |
| [2025-06-25 17:55:38,178][03778] Updated weights for policy 0, policy_version 540 (0.0022) | |
| [2025-06-25 17:55:39,648][03575] Fps is (10 sec: 3686.3, 60 sec: 4096.0, 300 sec: 3998.8). Total num frames: 2215936. Throughput: 0: 1025.5. Samples: 554300. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:55:39,649][03575] Avg episode reward: [(0, '13.942')] | |
| [2025-06-25 17:55:39,654][03764] Saving new best policy, reward=13.942! | |
| [2025-06-25 17:55:44,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4026.6). Total num frames: 2240512. Throughput: 0: 1036.6. Samples: 557738. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:55:44,651][03575] Avg episode reward: [(0, '14.794')] | |
| [2025-06-25 17:55:44,657][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000547_2240512.pth... | |
| [2025-06-25 17:55:44,753][03764] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000310_1269760.pth | |
| [2025-06-25 17:55:44,763][03764] Saving new best policy, reward=14.794! | |
| [2025-06-25 17:55:46,996][03778] Updated weights for policy 0, policy_version 550 (0.0016) | |
| [2025-06-25 17:55:49,647][03575] Fps is (10 sec: 4505.8, 60 sec: 4096.0, 300 sec: 4026.6). Total num frames: 2260992. Throughput: 0: 1039.4. Samples: 564800. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:55:49,648][03575] Avg episode reward: [(0, '15.926')] | |
| [2025-06-25 17:55:49,654][03764] Saving new best policy, reward=15.926! | |
| [2025-06-25 17:55:54,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4012.7). Total num frames: 2277376. Throughput: 0: 1031.5. Samples: 569820. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:55:54,653][03575] Avg episode reward: [(0, '15.868')] | |
| [2025-06-25 17:55:57,237][03778] Updated weights for policy 0, policy_version 560 (0.0025) | |
| [2025-06-25 17:55:59,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4040.5). Total num frames: 2301952. Throughput: 0: 1040.7. Samples: 573396. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:55:59,649][03575] Avg episode reward: [(0, '16.702')] | |
| [2025-06-25 17:55:59,656][03764] Saving new best policy, reward=16.702! | |
| [2025-06-25 17:56:04,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4026.6). Total num frames: 2322432. Throughput: 0: 1034.8. Samples: 580222. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:56:04,649][03575] Avg episode reward: [(0, '16.471')] | |
| [2025-06-25 17:56:07,811][03778] Updated weights for policy 0, policy_version 570 (0.0017) | |
| [2025-06-25 17:56:09,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4026.6). Total num frames: 2342912. Throughput: 0: 1035.2. Samples: 585390. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:56:09,653][03575] Avg episode reward: [(0, '16.604')] | |
| [2025-06-25 17:56:14,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4232.5, 300 sec: 4054.3). Total num frames: 2367488. Throughput: 0: 1037.9. Samples: 588952. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:56:14,652][03575] Avg episode reward: [(0, '16.273')] | |
| [2025-06-25 17:56:16,311][03778] Updated weights for policy 0, policy_version 580 (0.0025) | |
| [2025-06-25 17:56:19,648][03575] Fps is (10 sec: 4095.6, 60 sec: 4095.9, 300 sec: 4040.4). Total num frames: 2383872. Throughput: 0: 1031.9. Samples: 595668. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:56:19,654][03575] Avg episode reward: [(0, '16.027')] | |
| [2025-06-25 17:56:24,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4040.5). Total num frames: 2404352. Throughput: 0: 1040.0. Samples: 601100. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:56:24,655][03575] Avg episode reward: [(0, '17.332')] | |
| [2025-06-25 17:56:24,660][03764] Saving new best policy, reward=17.332! | |
| [2025-06-25 17:56:26,906][03778] Updated weights for policy 0, policy_version 590 (0.0020) | |
| [2025-06-25 17:56:29,647][03575] Fps is (10 sec: 4506.0, 60 sec: 4164.3, 300 sec: 4068.2). Total num frames: 2428928. Throughput: 0: 1040.7. Samples: 604570. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:56:29,651][03575] Avg episode reward: [(0, '17.832')] | |
| [2025-06-25 17:56:29,654][03764] Saving new best policy, reward=17.832! | |
| [2025-06-25 17:56:34,650][03575] Fps is (10 sec: 4094.8, 60 sec: 4095.8, 300 sec: 4054.3). Total num frames: 2445312. Throughput: 0: 1021.3. Samples: 610762. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:56:34,652][03575] Avg episode reward: [(0, '18.160')] | |
| [2025-06-25 17:56:34,661][03764] Saving new best policy, reward=18.160! | |
| [2025-06-25 17:56:37,548][03778] Updated weights for policy 0, policy_version 600 (0.0016) | |
| [2025-06-25 17:56:39,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4054.4). Total num frames: 2465792. Throughput: 0: 1034.8. Samples: 616384. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:56:39,652][03575] Avg episode reward: [(0, '18.313')] | |
| [2025-06-25 17:56:39,657][03764] Saving new best policy, reward=18.313! | |
| [2025-06-25 17:56:44,647][03575] Fps is (10 sec: 4506.9, 60 sec: 4164.3, 300 sec: 4082.1). Total num frames: 2490368. Throughput: 0: 1031.0. Samples: 619792. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:56:44,649][03575] Avg episode reward: [(0, '18.398')] | |
| [2025-06-25 17:56:44,658][03764] Saving new best policy, reward=18.398! | |
| [2025-06-25 17:56:46,644][03778] Updated weights for policy 0, policy_version 610 (0.0016) | |
| [2025-06-25 17:56:49,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 4054.3). Total num frames: 2506752. Throughput: 0: 1018.9. Samples: 626074. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:56:49,649][03575] Avg episode reward: [(0, '18.521')] | |
| [2025-06-25 17:56:49,652][03764] Saving new best policy, reward=18.521! | |
| [2025-06-25 17:56:54,647][03575] Fps is (10 sec: 3686.3, 60 sec: 4164.3, 300 sec: 4068.2). Total num frames: 2527232. Throughput: 0: 1030.6. Samples: 631768. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:56:54,649][03575] Avg episode reward: [(0, '17.783')] | |
| [2025-06-25 17:56:56,828][03778] Updated weights for policy 0, policy_version 620 (0.0012) | |
| [2025-06-25 17:56:59,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4164.3, 300 sec: 4096.0). Total num frames: 2551808. Throughput: 0: 1031.1. Samples: 635350. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:56:59,649][03575] Avg episode reward: [(0, '19.030')] | |
| [2025-06-25 17:56:59,654][03764] Saving new best policy, reward=19.030! | |
| [2025-06-25 17:57:04,647][03575] Fps is (10 sec: 4096.1, 60 sec: 4096.0, 300 sec: 4068.2). Total num frames: 2568192. Throughput: 0: 1014.5. Samples: 641318. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 17:57:04,649][03575] Avg episode reward: [(0, '19.217')] | |
| [2025-06-25 17:57:04,667][03764] Saving new best policy, reward=19.217! | |
| [2025-06-25 17:57:07,490][03778] Updated weights for policy 0, policy_version 630 (0.0019) | |
| [2025-06-25 17:57:09,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4082.1). Total num frames: 2588672. Throughput: 0: 1027.2. Samples: 647324. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:57:09,649][03575] Avg episode reward: [(0, '19.383')] | |
| [2025-06-25 17:57:09,651][03764] Saving new best policy, reward=19.383! | |
| [2025-06-25 17:57:14,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4096.0). Total num frames: 2613248. Throughput: 0: 1028.6. Samples: 650856. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:57:14,649][03575] Avg episode reward: [(0, '18.525')] | |
| [2025-06-25 17:57:16,625][03778] Updated weights for policy 0, policy_version 640 (0.0022) | |
| [2025-06-25 17:57:19,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.1, 300 sec: 4082.1). Total num frames: 2629632. Throughput: 0: 1019.8. Samples: 656652. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:57:19,649][03575] Avg episode reward: [(0, '17.107')] | |
| [2025-06-25 17:57:24,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4096.0). Total num frames: 2650112. Throughput: 0: 1035.6. Samples: 662984. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:57:24,649][03575] Avg episode reward: [(0, '16.025')] | |
| [2025-06-25 17:57:26,454][03778] Updated weights for policy 0, policy_version 650 (0.0019) | |
| [2025-06-25 17:57:29,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4109.9). Total num frames: 2674688. Throughput: 0: 1038.5. Samples: 666526. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:57:29,649][03575] Avg episode reward: [(0, '15.069')] | |
| [2025-06-25 17:57:34,647][03575] Fps is (10 sec: 4095.9, 60 sec: 4096.2, 300 sec: 4082.1). Total num frames: 2691072. Throughput: 0: 1022.0. Samples: 672066. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:57:34,649][03575] Avg episode reward: [(0, '15.833')] | |
| [2025-06-25 17:57:36,881][03778] Updated weights for policy 0, policy_version 660 (0.0020) | |
| [2025-06-25 17:57:39,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4109.9). Total num frames: 2715648. Throughput: 0: 1041.4. Samples: 678632. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:57:39,653][03575] Avg episode reward: [(0, '17.245')] | |
| [2025-06-25 17:57:44,647][03575] Fps is (10 sec: 4505.7, 60 sec: 4096.0, 300 sec: 4109.9). Total num frames: 2736128. Throughput: 0: 1040.7. Samples: 682182. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:57:44,649][03575] Avg episode reward: [(0, '17.360')] | |
| [2025-06-25 17:57:44,669][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000668_2736128.pth... | |
| [2025-06-25 17:57:44,805][03764] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000427_1748992.pth | |
| [2025-06-25 17:57:46,735][03778] Updated weights for policy 0, policy_version 670 (0.0023) | |
| [2025-06-25 17:57:49,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4082.1). Total num frames: 2752512. Throughput: 0: 1022.7. Samples: 687338. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:57:49,649][03575] Avg episode reward: [(0, '17.622')] | |
| [2025-06-25 17:57:54,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4109.9). Total num frames: 2777088. Throughput: 0: 1035.3. Samples: 693914. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) | |
| [2025-06-25 17:57:54,652][03575] Avg episode reward: [(0, '17.364')] | |
| [2025-06-25 17:57:56,194][03778] Updated weights for policy 0, policy_version 680 (0.0023) | |
| [2025-06-25 17:57:59,650][03575] Fps is (10 sec: 4504.3, 60 sec: 4095.8, 300 sec: 4109.8). Total num frames: 2797568. Throughput: 0: 1036.9. Samples: 697520. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) | |
| [2025-06-25 17:57:59,654][03575] Avg episode reward: [(0, '17.154')] | |
| [2025-06-25 17:58:04,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4082.1). Total num frames: 2813952. Throughput: 0: 1020.4. Samples: 702568. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:58:04,649][03575] Avg episode reward: [(0, '16.938')] | |
| [2025-06-25 17:58:06,779][03778] Updated weights for policy 0, policy_version 690 (0.0023) | |
| [2025-06-25 17:58:09,647][03575] Fps is (10 sec: 4097.2, 60 sec: 4164.3, 300 sec: 4109.9). Total num frames: 2838528. Throughput: 0: 1033.6. Samples: 709494. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:58:09,649][03575] Avg episode reward: [(0, '17.813')] | |
| [2025-06-25 17:58:14,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4096.0). Total num frames: 2859008. Throughput: 0: 1033.7. Samples: 713044. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:58:14,663][03575] Avg episode reward: [(0, '16.802')] | |
| [2025-06-25 17:58:16,903][03778] Updated weights for policy 0, policy_version 700 (0.0029) | |
| [2025-06-25 17:58:19,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4082.1). Total num frames: 2875392. Throughput: 0: 1023.5. Samples: 718122. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:58:19,649][03575] Avg episode reward: [(0, '16.460')] | |
| [2025-06-25 17:58:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4109.9). Total num frames: 2899968. Throughput: 0: 1037.0. Samples: 725296. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 17:58:24,649][03575] Avg episode reward: [(0, '18.770')] | |
| [2025-06-25 17:58:25,734][03778] Updated weights for policy 0, policy_version 710 (0.0017) | |
| [2025-06-25 17:58:29,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4096.0). Total num frames: 2920448. Throughput: 0: 1036.7. Samples: 728834. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:58:29,649][03575] Avg episode reward: [(0, '18.430')] | |
| [2025-06-25 17:58:34,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4096.0). Total num frames: 2940928. Throughput: 0: 1032.8. Samples: 733812. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:58:34,649][03575] Avg episode reward: [(0, '20.444')] | |
| [2025-06-25 17:58:34,657][03764] Saving new best policy, reward=20.444! | |
| [2025-06-25 17:58:36,059][03778] Updated weights for policy 0, policy_version 720 (0.0028) | |
| [2025-06-25 17:58:39,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4164.3, 300 sec: 4123.8). Total num frames: 2965504. Throughput: 0: 1043.9. Samples: 740888. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:58:39,649][03575] Avg episode reward: [(0, '20.895')] | |
| [2025-06-25 17:58:39,650][03764] Saving new best policy, reward=20.895! | |
| [2025-06-25 17:58:44,648][03575] Fps is (10 sec: 4095.7, 60 sec: 4096.0, 300 sec: 4096.0). Total num frames: 2981888. Throughput: 0: 1040.9. Samples: 744356. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:58:44,652][03575] Avg episode reward: [(0, '20.597')] | |
| [2025-06-25 17:58:46,586][03778] Updated weights for policy 0, policy_version 730 (0.0024) | |
| [2025-06-25 17:58:49,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4096.0). Total num frames: 3002368. Throughput: 0: 1044.1. Samples: 749554. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:58:49,651][03575] Avg episode reward: [(0, '19.728')] | |
| [2025-06-25 17:58:54,647][03575] Fps is (10 sec: 4505.9, 60 sec: 4164.3, 300 sec: 4123.8). Total num frames: 3026944. Throughput: 0: 1048.4. Samples: 756670. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:58:54,648][03575] Avg episode reward: [(0, '18.923')] | |
| [2025-06-25 17:58:55,048][03778] Updated weights for policy 0, policy_version 740 (0.0018) | |
| [2025-06-25 17:58:59,648][03575] Fps is (10 sec: 4095.6, 60 sec: 4096.1, 300 sec: 4109.9). Total num frames: 3043328. Throughput: 0: 1044.9. Samples: 760066. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:58:59,652][03575] Avg episode reward: [(0, '18.622')] | |
| [2025-06-25 17:59:04,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4232.5, 300 sec: 4123.8). Total num frames: 3067904. Throughput: 0: 1049.6. Samples: 765352. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:59:04,649][03575] Avg episode reward: [(0, '19.524')] | |
| [2025-06-25 17:59:05,460][03778] Updated weights for policy 0, policy_version 750 (0.0018) | |
| [2025-06-25 17:59:09,647][03575] Fps is (10 sec: 4506.1, 60 sec: 4164.3, 300 sec: 4123.8). Total num frames: 3088384. Throughput: 0: 1048.8. Samples: 772492. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:59:09,652][03575] Avg episode reward: [(0, '22.342')] | |
| [2025-06-25 17:59:09,708][03764] Saving new best policy, reward=22.342! | |
| [2025-06-25 17:59:14,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4096.0, 300 sec: 4109.9). Total num frames: 3104768. Throughput: 0: 1037.2. Samples: 775508. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:59:14,653][03575] Avg episode reward: [(0, '21.661')] | |
| [2025-06-25 17:59:15,815][03778] Updated weights for policy 0, policy_version 760 (0.0014) | |
| [2025-06-25 17:59:19,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4232.5, 300 sec: 4123.8). Total num frames: 3129344. Throughput: 0: 1051.3. Samples: 781120. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:59:19,653][03575] Avg episode reward: [(0, '21.732')] | |
| [2025-06-25 17:59:24,219][03778] Updated weights for policy 0, policy_version 770 (0.0018) | |
| [2025-06-25 17:59:24,647][03575] Fps is (10 sec: 4915.2, 60 sec: 4232.5, 300 sec: 4137.7). Total num frames: 3153920. Throughput: 0: 1054.7. Samples: 788350. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 17:59:24,652][03575] Avg episode reward: [(0, '21.231')] | |
| [2025-06-25 17:59:29,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4123.8). Total num frames: 3170304. Throughput: 0: 1040.4. Samples: 791172. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:59:29,651][03575] Avg episode reward: [(0, '19.802')] | |
| [2025-06-25 17:59:34,572][03778] Updated weights for policy 0, policy_version 780 (0.0018) | |
| [2025-06-25 17:59:34,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4232.5, 300 sec: 4151.5). Total num frames: 3194880. Throughput: 0: 1056.1. Samples: 797078. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:59:34,649][03575] Avg episode reward: [(0, '19.553')] | |
| [2025-06-25 17:59:39,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4164.3, 300 sec: 4151.5). Total num frames: 3215360. Throughput: 0: 1056.5. Samples: 804212. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 17:59:39,649][03575] Avg episode reward: [(0, '21.310')] | |
| [2025-06-25 17:59:44,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4123.8). Total num frames: 3231744. Throughput: 0: 1033.9. Samples: 806590. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:59:44,649][03575] Avg episode reward: [(0, '21.116')] | |
| [2025-06-25 17:59:44,658][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000789_3231744.pth... | |
| [2025-06-25 17:59:44,769][03764] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000547_2240512.pth | |
| [2025-06-25 17:59:44,861][03778] Updated weights for policy 0, policy_version 790 (0.0012) | |
| [2025-06-25 17:59:49,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4232.5, 300 sec: 4151.5). Total num frames: 3256320. Throughput: 0: 1055.7. Samples: 812860. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:59:49,649][03575] Avg episode reward: [(0, '21.615')] | |
| [2025-06-25 17:59:53,551][03778] Updated weights for policy 0, policy_version 800 (0.0027) | |
| [2025-06-25 17:59:54,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4164.3, 300 sec: 4151.5). Total num frames: 3276800. Throughput: 0: 1056.0. Samples: 820010. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 17:59:54,650][03575] Avg episode reward: [(0, '22.373')] | |
| [2025-06-25 17:59:54,711][03764] Saving new best policy, reward=22.373! | |
| [2025-06-25 17:59:59,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4232.6, 300 sec: 4137.7). Total num frames: 3297280. Throughput: 0: 1036.1. Samples: 822132. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 17:59:59,649][03575] Avg episode reward: [(0, '22.233')] | |
| [2025-06-25 18:00:03,838][03778] Updated weights for policy 0, policy_version 810 (0.0019) | |
| [2025-06-25 18:00:04,648][03575] Fps is (10 sec: 4095.8, 60 sec: 4164.2, 300 sec: 4151.5). Total num frames: 3317760. Throughput: 0: 1055.9. Samples: 828636. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:00:04,649][03575] Avg episode reward: [(0, '21.068')] | |
| [2025-06-25 18:00:09,653][03575] Fps is (10 sec: 4502.8, 60 sec: 4232.1, 300 sec: 4165.3). Total num frames: 3342336. Throughput: 0: 1044.3. Samples: 835348. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 18:00:09,655][03575] Avg episode reward: [(0, '22.632')] | |
| [2025-06-25 18:00:09,660][03764] Saving new best policy, reward=22.632! | |
| [2025-06-25 18:00:14,373][03778] Updated weights for policy 0, policy_version 820 (0.0025) | |
| [2025-06-25 18:00:14,647][03575] Fps is (10 sec: 4096.2, 60 sec: 4232.5, 300 sec: 4137.7). Total num frames: 3358720. Throughput: 0: 1029.2. Samples: 837488. Policy #0 lag: (min: 0.0, avg: 0.6, max: 1.0) | |
| [2025-06-25 18:00:14,649][03575] Avg episode reward: [(0, '22.324')] | |
| [2025-06-25 18:00:19,647][03575] Fps is (10 sec: 4098.4, 60 sec: 4232.5, 300 sec: 4165.4). Total num frames: 3383296. Throughput: 0: 1046.2. Samples: 844158. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 18:00:19,650][03575] Avg episode reward: [(0, '22.844')] | |
| [2025-06-25 18:00:19,655][03764] Saving new best policy, reward=22.844! | |
| [2025-06-25 18:00:23,355][03778] Updated weights for policy 0, policy_version 830 (0.0020) | |
| [2025-06-25 18:00:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4096.0, 300 sec: 4137.7). Total num frames: 3399680. Throughput: 0: 1034.9. Samples: 850784. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 18:00:24,649][03575] Avg episode reward: [(0, '21.506')] | |
| [2025-06-25 18:00:29,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4137.7). Total num frames: 3420160. Throughput: 0: 1030.2. Samples: 852948. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:00:29,652][03575] Avg episode reward: [(0, '20.109')] | |
| [2025-06-25 18:00:33,323][03778] Updated weights for policy 0, policy_version 840 (0.0017) | |
| [2025-06-25 18:00:34,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4164.3, 300 sec: 4165.4). Total num frames: 3444736. Throughput: 0: 1045.0. Samples: 859886. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:00:34,649][03575] Avg episode reward: [(0, '19.428')] | |
| [2025-06-25 18:00:39,647][03575] Fps is (10 sec: 4096.1, 60 sec: 4096.0, 300 sec: 4137.7). Total num frames: 3461120. Throughput: 0: 1026.1. Samples: 866186. Policy #0 lag: (min: 0.0, avg: 0.2, max: 1.0) | |
| [2025-06-25 18:00:39,649][03575] Avg episode reward: [(0, '19.119')] | |
| [2025-06-25 18:00:43,630][03778] Updated weights for policy 0, policy_version 850 (0.0014) | |
| [2025-06-25 18:00:44,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4232.5, 300 sec: 4151.5). Total num frames: 3485696. Throughput: 0: 1031.1. Samples: 868530. Policy #0 lag: (min: 0.0, avg: 0.3, max: 2.0) | |
| [2025-06-25 18:00:44,649][03575] Avg episode reward: [(0, '21.217')] | |
| [2025-06-25 18:00:49,650][03575] Fps is (10 sec: 4504.3, 60 sec: 4164.1, 300 sec: 4165.4). Total num frames: 3506176. Throughput: 0: 1046.7. Samples: 875738. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 18:00:49,651][03575] Avg episode reward: [(0, '23.556')] | |
| [2025-06-25 18:00:49,726][03764] Saving new best policy, reward=23.556! | |
| [2025-06-25 18:00:53,134][03778] Updated weights for policy 0, policy_version 860 (0.0014) | |
| [2025-06-25 18:00:54,649][03575] Fps is (10 sec: 4095.2, 60 sec: 4164.1, 300 sec: 4151.5). Total num frames: 3526656. Throughput: 0: 1031.5. Samples: 881760. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 18:00:54,651][03575] Avg episode reward: [(0, '23.618')] | |
| [2025-06-25 18:00:54,669][03764] Saving new best policy, reward=23.618! | |
| [2025-06-25 18:00:59,647][03575] Fps is (10 sec: 4097.2, 60 sec: 4164.3, 300 sec: 4151.5). Total num frames: 3547136. Throughput: 0: 1039.1. Samples: 884246. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 18:00:59,651][03575] Avg episode reward: [(0, '24.900')] | |
| [2025-06-25 18:00:59,654][03764] Saving new best policy, reward=24.900! | |
| [2025-06-25 18:01:02,889][03778] Updated weights for policy 0, policy_version 870 (0.0017) | |
| [2025-06-25 18:01:04,647][03575] Fps is (10 sec: 4506.5, 60 sec: 4232.6, 300 sec: 4165.4). Total num frames: 3571712. Throughput: 0: 1048.6. Samples: 891344. Policy #0 lag: (min: 0.0, avg: 0.6, max: 2.0) | |
| [2025-06-25 18:01:04,650][03575] Avg episode reward: [(0, '25.067')] | |
| [2025-06-25 18:01:04,656][03764] Saving new best policy, reward=25.067! | |
| [2025-06-25 18:01:09,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4028.1, 300 sec: 4123.8). Total num frames: 3584000. Throughput: 0: 1020.8. Samples: 896718. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 18:01:09,651][03575] Avg episode reward: [(0, '26.615')] | |
| [2025-06-25 18:01:09,746][03764] Saving new best policy, reward=26.615! | |
| [2025-06-25 18:01:13,736][03778] Updated weights for policy 0, policy_version 880 (0.0018) | |
| [2025-06-25 18:01:14,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4151.6). Total num frames: 3608576. Throughput: 0: 1034.2. Samples: 899486. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 18:01:14,651][03575] Avg episode reward: [(0, '26.994')] | |
| [2025-06-25 18:01:14,661][03764] Saving new best policy, reward=26.994! | |
| [2025-06-25 18:01:19,647][03575] Fps is (10 sec: 4505.6, 60 sec: 4096.0, 300 sec: 4151.5). Total num frames: 3629056. Throughput: 0: 1034.8. Samples: 906452. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 18:01:19,651][03575] Avg episode reward: [(0, '27.363')] | |
| [2025-06-25 18:01:19,687][03764] Saving new best policy, reward=27.363! | |
| [2025-06-25 18:01:23,987][03778] Updated weights for policy 0, policy_version 890 (0.0018) | |
| [2025-06-25 18:01:24,648][03575] Fps is (10 sec: 3686.3, 60 sec: 4096.0, 300 sec: 4123.8). Total num frames: 3645440. Throughput: 0: 1015.6. Samples: 911890. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 18:01:24,652][03575] Avg episode reward: [(0, '28.622')] | |
| [2025-06-25 18:01:24,658][03764] Saving new best policy, reward=28.622! | |
| [2025-06-25 18:01:29,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4164.3, 300 sec: 4151.6). Total num frames: 3670016. Throughput: 0: 1033.2. Samples: 915024. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 18:01:29,650][03575] Avg episode reward: [(0, '27.653')] | |
| [2025-06-25 18:01:32,812][03778] Updated weights for policy 0, policy_version 900 (0.0019) | |
| [2025-06-25 18:01:34,648][03575] Fps is (10 sec: 4915.0, 60 sec: 4164.2, 300 sec: 4165.4). Total num frames: 3694592. Throughput: 0: 1032.5. Samples: 922200. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:01:34,650][03575] Avg episode reward: [(0, '27.259')] | |
| [2025-06-25 18:01:39,648][03575] Fps is (10 sec: 4095.7, 60 sec: 4164.2, 300 sec: 4137.6). Total num frames: 3710976. Throughput: 0: 1011.6. Samples: 927282. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:01:39,652][03575] Avg episode reward: [(0, '25.535')] | |
| [2025-06-25 18:01:43,013][03778] Updated weights for policy 0, policy_version 910 (0.0040) | |
| [2025-06-25 18:01:44,650][03575] Fps is (10 sec: 3685.9, 60 sec: 4095.8, 300 sec: 4151.5). Total num frames: 3731456. Throughput: 0: 1033.3. Samples: 930746. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:01:44,651][03575] Avg episode reward: [(0, '22.998')] | |
| [2025-06-25 18:01:44,716][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000912_3735552.pth... | |
| [2025-06-25 18:01:44,839][03764] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000668_2736128.pth | |
| [2025-06-25 18:01:49,651][03575] Fps is (10 sec: 4504.2, 60 sec: 4164.2, 300 sec: 4165.4). Total num frames: 3756032. Throughput: 0: 1034.6. Samples: 937906. Policy #0 lag: (min: 0.0, avg: 0.3, max: 1.0) | |
| [2025-06-25 18:01:49,652][03575] Avg episode reward: [(0, '21.539')] | |
| [2025-06-25 18:01:53,279][03778] Updated weights for policy 0, policy_version 920 (0.0022) | |
| [2025-06-25 18:01:54,647][03575] Fps is (10 sec: 4096.9, 60 sec: 4096.1, 300 sec: 4137.7). Total num frames: 3772416. Throughput: 0: 1027.4. Samples: 942950. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:01:54,652][03575] Avg episode reward: [(0, '22.926')] | |
| [2025-06-25 18:01:59,647][03575] Fps is (10 sec: 4097.6, 60 sec: 4164.3, 300 sec: 4165.4). Total num frames: 3796992. Throughput: 0: 1045.7. Samples: 946542. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:01:59,649][03575] Avg episode reward: [(0, '22.170')] | |
| [2025-06-25 18:02:02,018][03778] Updated weights for policy 0, policy_version 930 (0.0027) | |
| [2025-06-25 18:02:04,647][03575] Fps is (10 sec: 4505.5, 60 sec: 4096.0, 300 sec: 4165.4). Total num frames: 3817472. Throughput: 0: 1049.2. Samples: 953668. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 18:02:04,651][03575] Avg episode reward: [(0, '22.828')] | |
| [2025-06-25 18:02:09,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4137.7). Total num frames: 3833856. Throughput: 0: 1037.1. Samples: 958558. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 18:02:09,649][03575] Avg episode reward: [(0, '23.557')] | |
| [2025-06-25 18:02:12,525][03778] Updated weights for policy 0, policy_version 940 (0.0022) | |
| [2025-06-25 18:02:14,647][03575] Fps is (10 sec: 4096.1, 60 sec: 4164.3, 300 sec: 4165.4). Total num frames: 3858432. Throughput: 0: 1047.6. Samples: 962166. Policy #0 lag: (min: 0.0, avg: 0.5, max: 2.0) | |
| [2025-06-25 18:02:14,649][03575] Avg episode reward: [(0, '24.953')] | |
| [2025-06-25 18:02:19,649][03575] Fps is (10 sec: 4504.8, 60 sec: 4164.1, 300 sec: 4165.4). Total num frames: 3878912. Throughput: 0: 1047.3. Samples: 969330. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 18:02:19,650][03575] Avg episode reward: [(0, '24.493')] | |
| [2025-06-25 18:02:22,770][03778] Updated weights for policy 0, policy_version 950 (0.0015) | |
| [2025-06-25 18:02:24,647][03575] Fps is (10 sec: 4096.0, 60 sec: 4232.6, 300 sec: 4151.5). Total num frames: 3899392. Throughput: 0: 1049.4. Samples: 974506. Policy #0 lag: (min: 0.0, avg: 0.5, max: 1.0) | |
| [2025-06-25 18:02:24,649][03575] Avg episode reward: [(0, '24.836')] | |
| [2025-06-25 18:02:29,647][03575] Fps is (10 sec: 4506.5, 60 sec: 4232.5, 300 sec: 4179.3). Total num frames: 3923968. Throughput: 0: 1052.9. Samples: 978124. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:02:29,649][03575] Avg episode reward: [(0, '25.251')] | |
| [2025-06-25 18:02:31,253][03778] Updated weights for policy 0, policy_version 960 (0.0022) | |
| [2025-06-25 18:02:34,648][03575] Fps is (10 sec: 4095.8, 60 sec: 4096.0, 300 sec: 4151.5). Total num frames: 3940352. Throughput: 0: 1050.8. Samples: 985190. Policy #0 lag: (min: 0.0, avg: 0.4, max: 2.0) | |
| [2025-06-25 18:02:34,651][03575] Avg episode reward: [(0, '24.470')] | |
| [2025-06-25 18:02:39,647][03575] Fps is (10 sec: 3686.4, 60 sec: 4164.3, 300 sec: 4151.5). Total num frames: 3960832. Throughput: 0: 1053.0. Samples: 990336. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:02:39,653][03575] Avg episode reward: [(0, '23.719')] | |
| [2025-06-25 18:02:41,631][03778] Updated weights for policy 0, policy_version 970 (0.0031) | |
| [2025-06-25 18:02:44,647][03575] Fps is (10 sec: 4505.8, 60 sec: 4232.7, 300 sec: 4179.3). Total num frames: 3985408. Throughput: 0: 1053.2. Samples: 993934. Policy #0 lag: (min: 0.0, avg: 0.4, max: 1.0) | |
| [2025-06-25 18:02:44,649][03575] Avg episode reward: [(0, '24.160')] | |
| [2025-06-25 18:02:49,231][03575] Component Batcher_0 stopped! | |
| [2025-06-25 18:02:49,235][03575] Component RolloutWorker_w2 process died already! Don't wait for it. | |
| [2025-06-25 18:02:49,230][03764] Stopping Batcher_0... | |
| [2025-06-25 18:02:49,237][03764] Loop batcher_evt_loop terminating... | |
| [2025-06-25 18:02:49,238][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:02:49,354][03778] Weights refcount: 2 0 | |
| [2025-06-25 18:02:49,360][03575] Component InferenceWorker_p0-w0 stopped! | |
| [2025-06-25 18:02:49,365][03778] Stopping InferenceWorker_p0-w0... | |
| [2025-06-25 18:02:49,366][03778] Loop inference_proc0-0_evt_loop terminating... | |
| [2025-06-25 18:02:49,368][03764] Removing /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000789_3231744.pth | |
| [2025-06-25 18:02:49,379][03764] Saving /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:02:49,548][03575] Component LearnerWorker_p0 stopped! | |
| [2025-06-25 18:02:49,551][03764] Stopping LearnerWorker_p0... | |
| [2025-06-25 18:02:49,555][03764] Loop learner_proc0_evt_loop terminating... | |
| [2025-06-25 18:02:49,759][03575] Component RolloutWorker_w0 stopped! | |
| [2025-06-25 18:02:49,765][03779] Stopping RolloutWorker_w0... | |
| [2025-06-25 18:02:49,766][03779] Loop rollout_proc0_evt_loop terminating... | |
| [2025-06-25 18:02:49,795][03575] Component RolloutWorker_w6 stopped! | |
| [2025-06-25 18:02:49,800][03784] Stopping RolloutWorker_w6... | |
| [2025-06-25 18:02:49,801][03784] Loop rollout_proc6_evt_loop terminating... | |
| [2025-06-25 18:02:49,820][03575] Component RolloutWorker_w4 stopped! | |
| [2025-06-25 18:02:49,824][03782] Stopping RolloutWorker_w4... | |
| [2025-06-25 18:02:49,824][03782] Loop rollout_proc4_evt_loop terminating... | |
| [2025-06-25 18:02:49,870][03785] Stopping RolloutWorker_w7... | |
| [2025-06-25 18:02:49,870][03785] Loop rollout_proc7_evt_loop terminating... | |
| [2025-06-25 18:02:49,870][03575] Component RolloutWorker_w7 stopped! | |
| [2025-06-25 18:02:49,897][03575] Component RolloutWorker_w1 stopped! | |
| [2025-06-25 18:02:49,895][03777] Stopping RolloutWorker_w1... | |
| [2025-06-25 18:02:49,899][03777] Loop rollout_proc1_evt_loop terminating... | |
| [2025-06-25 18:02:49,912][03783] Stopping RolloutWorker_w5... | |
| [2025-06-25 18:02:49,912][03575] Component RolloutWorker_w5 stopped! | |
| [2025-06-25 18:02:49,913][03783] Loop rollout_proc5_evt_loop terminating... | |
| [2025-06-25 18:02:49,928][03781] Stopping RolloutWorker_w3... | |
| [2025-06-25 18:02:49,928][03781] Loop rollout_proc3_evt_loop terminating... | |
| [2025-06-25 18:02:49,927][03575] Component RolloutWorker_w3 stopped! | |
| [2025-06-25 18:02:49,933][03575] Waiting for process learner_proc0 to stop... | |
| [2025-06-25 18:02:51,556][03575] Waiting for process inference_proc0-0 to join... | |
| [2025-06-25 18:02:51,557][03575] Waiting for process rollout_proc0 to join... | |
| [2025-06-25 18:02:53,422][03575] Waiting for process rollout_proc1 to join... | |
| [2025-06-25 18:02:53,677][03575] Waiting for process rollout_proc2 to join... | |
| [2025-06-25 18:02:53,679][03575] Waiting for process rollout_proc3 to join... | |
| [2025-06-25 18:02:53,681][03575] Waiting for process rollout_proc4 to join... | |
| [2025-06-25 18:02:53,682][03575] Waiting for process rollout_proc5 to join... | |
| [2025-06-25 18:02:53,684][03575] Waiting for process rollout_proc6 to join... | |
| [2025-06-25 18:02:53,686][03575] Waiting for process rollout_proc7 to join... | |
| [2025-06-25 18:02:53,687][03575] Batcher 0 profile tree view: | |
| batching: 24.0312, releasing_batches: 0.0241 | |
| [2025-06-25 18:02:53,688][03575] InferenceWorker_p0-w0 profile tree view: | |
| wait_policy: 0.0000 | |
| wait_policy_total: 392.5993 | |
| update_model: 8.6028 | |
| weight_update: 0.0012 | |
| one_step: 0.0177 | |
| handle_policy_step: 558.4357 | |
| deserialize: 13.4547, stack: 3.1980, obs_to_device_normalize: 121.3336, forward: 291.2990, send_messages: 24.7046 | |
| prepare_outputs: 81.0092 | |
| to_cpu: 50.4433 | |
| [2025-06-25 18:02:53,690][03575] Learner 0 profile tree view: | |
| misc: 0.0041, prepare_batch: 12.9368 | |
| train: 71.6136 | |
| epoch_init: 0.0103, minibatch_init: 0.0091, losses_postprocess: 0.6952, kl_divergence: 0.5889, after_optimizer: 33.0137 | |
| calculate_losses: 25.3158 | |
| losses_init: 0.0032, forward_head: 1.4279, bptt_initial: 16.7666, tail: 1.0660, advantages_returns: 0.2633, losses: 3.5503 | |
| bptt: 2.0012 | |
| bptt_forward_core: 1.9201 | |
| update: 11.4899 | |
| clip: 0.9878 | |
| [2025-06-25 18:02:53,691][03575] RolloutWorker_w0 profile tree view: | |
| wait_for_trajectories: 0.2701, enqueue_policy_requests: 113.3257, env_step: 768.7302, overhead: 11.6738, complete_rollouts: 6.3769 | |
| save_policy_outputs: 17.5510 | |
| split_output_tensors: 6.6227 | |
| [2025-06-25 18:02:53,692][03575] RolloutWorker_w7 profile tree view: | |
| wait_for_trajectories: 0.2597, enqueue_policy_requests: 92.2212, env_step: 788.9685, overhead: 12.7893, complete_rollouts: 7.3457 | |
| save_policy_outputs: 19.3153 | |
| split_output_tensors: 7.3965 | |
| [2025-06-25 18:02:53,693][03575] Loop Runner_EvtLoop terminating... | |
| [2025-06-25 18:02:53,694][03575] Runner profile tree view: | |
| main_loop: 1022.2904 | |
| [2025-06-25 18:02:53,695][03575] Collected {0: 4005888}, FPS: 3918.5 | |
| [2025-06-25 18:02:54,078][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:02:54,078][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:02:54,080][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:02:54,080][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:02:54,081][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:02:54,082][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:02:54,083][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:02:54,084][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:02:54,085][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:02:54,086][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:02:54,087][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:02:54,088][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:02:54,089][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:02:54,090][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:02:54,091][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:02:54,121][03575] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-06-25 18:02:54,123][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:02:54,125][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:02:54,138][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:02:54,234][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:02:54,234][03575] Policy head output size: 512 | |
| [2025-06-25 18:02:54,495][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:02:54,498][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:02:54,500][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:02:54,502][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:02:54,504][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:02:54,506][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:04:16,609][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:04:16,610][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:04:16,611][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:04:16,612][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:04:16,613][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:04:16,614][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:04:16,614][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:04:16,615][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:04:16,616][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:04:16,617][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:04:16,618][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:04:16,619][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:04:16,619][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:04:16,620][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:04:16,621][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:04:16,656][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:04:16,658][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:04:16,670][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:04:16,708][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:04:16,709][03575] Policy head output size: 512 | |
| [2025-06-25 18:04:16,726][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:04:16,727][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:04:16,729][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:04:16,730][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:04:16,732][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:04:16,734][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:07:56,862][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:07:56,868][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:07:56,870][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:07:56,871][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:07:56,874][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:07:56,875][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:07:56,875][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:07:56,876][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:07:56,877][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:07:56,877][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:07:56,881][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:07:56,882][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:07:56,883][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:07:56,883][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:07:56,884][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:07:57,031][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:07:57,046][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:07:57,095][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:07:57,340][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:07:57,341][03575] Policy head output size: 512 | |
| [2025-06-25 18:07:57,412][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:07:57,415][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:07:57,428][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:07:57,433][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:07:57,446][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:07:57,448][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:08:58,385][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:08:58,387][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:08:58,389][03575] Overriding arg 'load_checkpoint_kind' with value 'best' passed from command line | |
| [2025-06-25 18:08:58,391][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:08:58,392][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:08:58,393][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:08:58,395][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:08:58,396][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:08:58,397][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:08:58,398][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:08:58,399][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:08:58,400][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:08:58,401][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:08:58,402][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:08:58,403][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:08:58,407][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:08:58,502][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:08:58,511][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:08:58,551][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:08:58,702][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:08:58,716][03575] Policy head output size: 512 | |
| [2025-06-25 18:08:58,735][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/best_000000890_3645440_reward_28.622.pth... | |
| [2025-06-25 18:08:58,737][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:08:58,738][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/best_000000890_3645440_reward_28.622.pth... | |
| [2025-06-25 18:08:58,740][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:08:58,742][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/best_000000890_3645440_reward_28.622.pth... | |
| [2025-06-25 18:08:58,744][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:18,136][03575] Loading existing experiment configuration from ./train_dir/default_experiment/config.json | |
| [2025-06-25 18:09:18,139][03575] Overriding arg 'train_dir' with value './train_dir' passed from command line | |
| [2025-06-25 18:09:18,142][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:09:18,144][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:09:18,154][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:09:18,157][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:18,159][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:09:18,161][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:18,163][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:09:18,165][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:09:18,168][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:09:18,175][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:09:18,177][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:09:18,179][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:18,181][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:18,183][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:09:18,231][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:09:18,234][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:09:18,250][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:09:18,326][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:09:18,330][03575] Policy head output size: 512 | |
| [2025-06-25 18:09:18,369][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:18,374][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:18,379][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:18,382][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:18,385][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:18,389][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:25,364][03575] Loading existing experiment configuration from ./train_dir/default_experiment/config.json | |
| [2025-06-25 18:09:25,365][03575] Overriding arg 'train_dir' with value './train_dir' passed from command line | |
| [2025-06-25 18:09:25,367][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:09:25,368][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:09:25,370][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:09:25,371][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:25,372][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:09:25,374][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:25,375][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:09:25,377][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:09:25,377][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:09:25,378][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:09:25,379][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:09:25,380][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:25,380][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:25,381][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:09:25,429][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:09:25,430][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:09:25,481][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:09:25,611][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:09:25,612][03575] Policy head output size: 512 | |
| [2025-06-25 18:09:25,632][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:25,634][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:25,637][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:25,639][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:25,641][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:25,642][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:35,290][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:09:35,291][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:09:35,293][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:09:35,294][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:09:35,296][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:35,297][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:09:35,299][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:35,300][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:09:35,302][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:09:35,302][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:09:35,305][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:09:35,306][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:09:35,307][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:35,309][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:35,310][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:09:35,434][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:09:35,439][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:09:35,478][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:09:35,559][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:09:35,560][03575] Policy head output size: 512 | |
| [2025-06-25 18:09:35,578][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:35,579][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:35,581][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:35,582][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:35,584][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:35,585][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:58,166][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:09:58,167][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:09:58,167][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:09:58,168][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:09:58,169][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:58,170][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:09:58,171][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:09:58,172][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:09:58,173][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:09:58,174][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:09:58,175][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:09:58,175][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:09:58,176][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:58,177][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:09:58,178][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:09:58,205][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:09:58,206][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:09:58,216][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:09:58,253][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:09:58,254][03575] Policy head output size: 512 | |
| [2025-06-25 18:09:58,271][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:58,273][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:58,274][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:58,276][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:09:58,277][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:09:58,279][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:10:38,022][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:10:38,023][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:10:38,024][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:10:38,025][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:10:38,026][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:10:38,027][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:10:38,028][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:10:38,029][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:10:38,030][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:10:38,030][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:10:38,031][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:10:38,032][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:10:38,033][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:10:38,034][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:10:38,034][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:10:38,061][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:10:38,062][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:10:38,072][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:10:38,104][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:10:38,105][03575] Policy head output size: 512 | |
| [2025-06-25 18:10:38,122][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:10:38,125][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:10:38,126][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:10:38,128][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:10:38,129][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:10:38,131][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:12:25,992][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:12:26,020][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:12:26,020][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:12:26,022][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:12:26,024][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:12:26,027][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:12:26,030][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:12:26,032][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:12:26,035][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:12:26,037][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:12:26,039][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:12:26,044][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:12:26,049][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:12:26,050][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:12:26,051][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:12:26,139][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:12:26,150][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:12:26,178][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:12:26,260][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:12:26,261][03575] Policy head output size: 512 | |
| [2025-06-25 18:12:26,280][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:12:26,281][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:12:26,283][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:12:26,286][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:12:26,287][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:12:26,289][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:12:41,234][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:12:41,247][03575] Overriding arg 'env' with value 'doom_basic' passed from command line | |
| [2025-06-25 18:12:41,247][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:12:41,249][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:12:41,251][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:12:41,252][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:12:41,253][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:12:41,254][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:12:41,255][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:12:41,257][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:12:41,258][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:12:41,259][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:12:41,265][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:12:41,266][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:12:41,268][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:12:41,269][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:12:41,321][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:12:41,337][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:12:41,360][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:12:41,424][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:12:41,425][03575] Policy head output size: 512 | |
| [2025-06-25 18:12:41,457][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:12:41,459][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:12:41,461][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:12:41,463][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:12:41,464][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:12:41,466][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:15:00,560][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:15:00,560][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:15:00,562][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:15:00,563][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:15:00,564][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:15:00,565][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:15:00,565][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:15:00,566][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:15:00,567][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:15:00,568][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:15:00,569][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:15:00,570][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:15:00,571][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:15:00,571][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:15:00,573][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:15:00,601][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:15:00,602][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:15:00,612][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:15:00,645][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:15:00,646][03575] Policy head output size: 512 | |
| [2025-06-25 18:15:00,664][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:15:00,668][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:15:00,669][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:15:00,671][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:15:00,672][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:15:00,673][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:15:44,844][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:15:44,846][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:15:44,849][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:15:44,850][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:15:44,852][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:15:44,852][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:15:44,853][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:15:44,854][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:15:44,855][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:15:44,856][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:15:44,856][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:15:44,858][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:15:44,859][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:15:44,860][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:15:44,862][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:15:44,911][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:15:44,913][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:15:44,928][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:15:44,980][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:15:44,981][03575] Policy head output size: 512 | |
| [2025-06-25 18:15:45,008][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:15:45,010][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:15:45,012][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:15:45,014][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:15:45,015][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:15:45,017][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:16:46,185][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:16:46,186][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:16:46,187][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:16:46,188][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:16:46,189][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:16:46,190][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:16:46,191][03575] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! | |
| [2025-06-25 18:16:46,192][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:16:46,193][03575] Adding new argument 'push_to_hub'=True that is not in the saved config file! | |
| [2025-06-25 18:16:46,194][03575] Adding new argument 'hf_repository'='Cicikush/rl_course_vizdoom_health_gathering_supreme' that is not in the saved config file! | |
| [2025-06-25 18:16:46,195][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:16:46,196][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:16:46,197][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:16:46,198][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:16:46,199][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:16:46,226][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:16:46,228][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:16:46,237][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:16:46,271][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:16:46,271][03575] Policy head output size: 512 | |
| [2025-06-25 18:16:46,289][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:16:46,291][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:16:46,292][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:16:46,294][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:16:46,295][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:16:46,296][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:18:47,570][03575] Loading existing experiment configuration from /content/train_dir/default_experiment/config.json | |
| [2025-06-25 18:18:47,573][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:18:47,575][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:18:47,578][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:18:47,579][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:18:47,581][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:18:47,583][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:18:47,583][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:18:47,586][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:18:47,587][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:18:47,588][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:18:47,589][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:18:47,590][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:18:47,591][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:18:47,594][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:18:47,634][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:18:47,636][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:18:47,651][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:18:47,699][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:18:47,700][03575] Policy head output size: 512 | |
| [2025-06-25 18:18:47,726][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:18:47,728][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:18:47,729][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:18:47,731][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:18:47,732][03575] Loading state from checkpoint /content/train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:18:47,733][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:18:59,332][03575] Loading existing experiment configuration from ./train_dir/default_experiment/config.json | |
| [2025-06-25 18:18:59,333][03575] Overriding arg 'train_dir' with value './train_dir' passed from command line | |
| [2025-06-25 18:18:59,334][03575] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-06-25 18:18:59,336][03575] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-06-25 18:18:59,336][03575] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-06-25 18:18:59,337][03575] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:18:59,339][03575] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-06-25 18:18:59,341][03575] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-06-25 18:18:59,343][03575] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-06-25 18:18:59,346][03575] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-06-25 18:18:59,346][03575] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-06-25 18:18:59,347][03575] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-06-25 18:18:59,350][03575] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-06-25 18:18:59,351][03575] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-06-25 18:18:59,353][03575] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-06-25 18:18:59,355][03575] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-06-25 18:18:59,428][03575] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-06-25 18:18:59,432][03575] RunningMeanStd input shape: (1,) | |
| [2025-06-25 18:18:59,449][03575] ConvEncoder: input_channels=3 | |
| [2025-06-25 18:18:59,555][03575] Conv encoder output size: 512 | |
| [2025-06-25 18:18:59,560][03575] Policy head output size: 512 | |
| [2025-06-25 18:18:59,677][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:18:59,684][03575] Could not load from checkpoint, attempt 0 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:18:59,697][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:18:59,707][03575] Could not load from checkpoint, attempt 1 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |
| [2025-06-25 18:18:59,708][03575] Loading state from checkpoint ./train_dir/default_experiment/checkpoint_p0/checkpoint_000000978_4005888.pth... | |
| [2025-06-25 18:18:59,710][03575] Could not load from checkpoint, attempt 2 | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.11/dist-packages/sample_factory/algo/learning/learner.py", line 281, in load_checkpoint | |
| checkpoint_dict = torch.load(latest_checkpoint, map_location=device) | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| File "/usr/local/lib/python3.11/dist-packages/torch/serialization.py", line 1470, in load | |
| raise pickle.UnpicklingError(_get_wo_message(str(e))) from None | |
| _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. | |
| (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. | |
| (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. | |
| WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray.scalar was not an allowed global by default. Please use `torch.serialization.add_safe_globals([scalar])` or the `torch.serialization.safe_globals([scalar])` context manager to allowlist this global if you trust this class/function. | |
| Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html. | |