The program which used CUDA for computing in GPU reported error about memory:
Console
x
2
1
terminate called after throwing an instance of 'std::runtime_error'2
what(): [CUDA] an illegal memory access was encountered LightGBM/src/treelearner/cuda_tree_learner.cpp 239For common C++ program, we use gdb for debugging. For CUDA program, we should use cuda-gdb. Make sure to compile CUDA code with -g flag and then run:
Console
1
2
1
/usr/local/cuda-11.0/bin/cuda-gdb python32
(cuda-gdb) run test.pyAfter a while, we could see the exact memory corrupt position of the code:
Console
1
6
1
CUDA Exception: Warp Illegal Address2
The exception was triggered at PC 0x1668b2f0 (histogram_16_64_256.cu:182)3
Thread 1 "python3" received signal CUDA_EXCEPTION_14, Warp Illegal Address.4
[Switching focus to CUDA kernel 0, grid 10, block (2163,0,0), thread (0,0,0), device 0, sm 0, warp 3, lane 0]5
0x000000001668b380 in LightGBM::histogram16<<<(7360,1,1),(16,1,1)>>> () at LightGBM/src/treelearner/kernels/histogram_16_64_256.cu:1856
185 feature = (feature >> ((ind & 1) << 2)) & 0xf;