RuntimeError: CUDA error: no kernel image is available for execution on the deviceCUDA

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

3080 GPU 사용 도중, 파이토치 버전과 CUDA버전 상호 호환성 문제로 인하여 생긴 오류로 보인다.

+) 일단 anaconda 환경일 경우, cudatoolkit, pytorch를 제대로 설치한게 맞는지 conda list나 pip list로 내가 설치한 버전을  체크해보자.

Python 3.6.15 | packaged by conda-forge | (default, Dec  3 2021, 18:49:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True

상단처럼 torch.cuda.is_available()은 True라 떴는데,  아래와 같이 rand.cuda()함수에는 오류가 발생한다.

[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand(10).cuda()
/opt/conda/envs/[ANACONDA_ENV]/lib/python3.6/site-packages/torch/cuda/__init__.py:143: UserWarning: 
Graphics Device with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the Graphics Device GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/[ANACONDA_ENV]/lib/python3.6/site-packages/torch/_tensor.py", line 249, in __repr__
    return torch._tensor_str._str(self)
  File "/opt/conda/envs/[ANACONDA_ENV]/lib/python3.6/site-packages/torch/_tensor_str.py", line 415, in _str
    return _str_intern(self)
  File "/opt/conda/envs/[ANACONDA_ENV]/lib/python3.6/site-packages/torch/_tensor_str.py", line 390, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/opt/conda/envs/[ANACONDA_ENV]/lib/python3.6/site-packages/torch/_tensor_str.py", line 251, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/opt/conda/envs/[ANACONDA_ENV]/lib/python3.6/site-packages/torch/_tensor_str.py", line 90, in __init__
    nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

 

 

아래 사이트를 보고, 나에게 맞는 compute capability를 재설정해준 후, Pytorch를 재설치 한다.
(물론 이전에 설치한 pytorch는 삭제 후)

 

https://developer.nvidia.com/cuda-gpus#compute

CUDA GPUs - Compute Capability

상단 링크에서 각자 본인의 GPU와 맞는 값을 찾은 후, 아래 명령어로 설정해주자.

export TORCH_CUDA_ARCH_LIST=$COMPUTE_COMPABILITY

ex) export TORCH_CUDA_ARCH_LIST=8.6

 

 

삭제하고 재설치하는 명령어는 아래 pytorch공식 홈페이지를 참고하자.

 

https://pytorch.org/get-started/locally/

PyTorch

 

# 내 경우에 GPU는 3080, cudatoolkit은 11.3.1을 쓰고 있었다.
ex) conda install pytorch==1.12.1 torchvision==0.13.1 cudatoolkit=11.3.1 -c pytorch

 

이제 동일한 명령어를 쳤을 때, 올바르게 뜬다

Python 3.6.15 | packaged by conda-forge | (default, Dec  3 2021, 18:49:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand(10).cuda()
tensor([0.5354, 0.9879, 0.9010, 0.5497, 0.7680, 0.0444, 0.1607, 0.7694, 0.9544,
        0.3291], device='cuda:0')

 

---

 

 

 

+) 3090의 경우 다음 참고 : https://yjs-program.tistory.com/260

RTX 3090 Pytorch

 

---

 

도움을 주신 참고블로그

[error] RuntimeError: CUDA error: no kernel image is available for execution on the device