runtime error

Exit code: 1. Reason: 449, in result return self.__get_result() ~~~~~~~~~~~~~~~~~^^ File "/root/.pyenv/versions/3.13.12/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/root/.pyenv/versions/3.13.12/lib/python3.13/concurrent/futures/thread.py", line 59, in run result = self.fn(*self.args, **self.kwargs) File "/app/inference.py", line 193, in task return _generate_one(model_id, prompt, p, device, system_prompt) File "/app/inference.py", line 133, in _generate_one outputs = do_generate(gen_kwargs) File "/app/inference.py", line 129, in do_generate return model.generate(**inputs, **kwargs) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.13.12/lib/python3.13/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context return func(*args, **kwargs) File "/root/.pyenv/versions/3.13.12/lib/python3.13/site-packages/transformers/generation/utils.py", line 2668, in generate result = decoding_method( self, ...<5 lines>... **model_kwargs, ) File "/root/.pyenv/versions/3.13.12/lib/python3.13/site-packages/transformers/generation/utils.py", line 2869, in _sample while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.pyenv/versions/3.13.12/lib/python3.13/site-packages/transformers/generation/utils.py", line 2694, in _has_unfinished_sequences elif this_peer_finished: ^^^^^^^^^^^^^^^^^^ torch.AcceleratorError: CUDA error: device-side assert triggered Search for `cudaErrorAssert' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Container logs: