打开微信,使用扫一扫进入页面后,点击右上角菜单,
点击“发送给朋友”或“分享到朋友圈”完成分享
大家好,当我在MLU模式下运行量化后的模型推理,在output = self.model(images)[-1]句遇到:
fill_ Op cannot run on MLU device, start running on CPU!
copy_ Op cannot run on MLU device, start running on CPU!
警告并报错,请问这是否代表该操作MLU算子不支持?还是有数据未迁移至MLU?
谢谢!
警告以及报错的文字如下:
[WARNING][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/operators/op_methods.cpp][line:637][fill_][thread:140451381954304][process:84710]:
fill_ Op cannot run on MLU device, start running on CPU!
[ERROR][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/core/tensor_impl.cpp][line:866][cpu_data][thread:140451381954304][process:84710]:
Both cpu_storage and mlu_storage are not initialized!
Please check is there any invalid tensor operates such as:
output = input.cpu() or output = input.to("cpu") in pytorch model when doing mlu/mfus inference.
Can not call cpu_data on an empty tensor.
[WARNING][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/operators/op_methods.cpp][line:68][copy_][thread:140451381954304][process:84710]:
copy_ Op cannot run on MLU device, start running on CPU!
[ERROR][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/core/tensor_impl.cpp][line:866][cpu_data][thread:140451381954304][process:84710]:
Both cpu_storage and mlu_storage are not initialized!
Please check is there any invalid tensor operates such as:
output = input.cpu() or output = input.to("cpu") in pytorch model when doing mlu/mfus inference.
Traceback (most recent call last):
File "src/test.py", line 145, in
prefetch_test(opt)
File "src/test.py", line 80, in prefetch_test
ret = detector.run(pre_processed_images)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/detectors/ _detector.py", line 138, in run
output, dets, forward_time = self.process(images, return_time=True)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/detectors/ctdet.py", line 30, in process
output = self.model(images)[-1]
File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 473, in forward
x = self.dla_up(x)
File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 412, in forward
ida( s, len( s) -i - 2, len( s))
File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 385, in forward
s[i] = upsample(project( s[i]))
File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 356, in forward
x = self.conv(x)
File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/dcn_v2.py", line 128, in forward
self.deformable_groups)
File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/dcn_v2.py", line 31, in forward
ctx.deformable_groups)
RuntimeError: Can not call cpu_data on an empty tensor. (cpu_data at /workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/core/tensor_impl.cpp:870)
#0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits
#1: torch_mlu::MLUTensorImpl::cpu_data() + 0xef4 (0x7fbc027589b4 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#2: torch_mlu::copy_to_cpu(at::Tensor&, at::Tensor const&) + 0xd6 (0x7fbc027683f6 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#3: torch_mlu::OpMethods::copy_(at::Tensor&, at::Tensor const&, bool) + 0x5f (0x7fbc02c6280f in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#4: torch_mlu::CnmlOps::copy_(at::Tensor&, at::Tensor const&, bool) + 0x2dc (0x7fbc02a0b60c in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#5: torch_mlu::AtenMluType::copy_(at::Tensor&, at::Tensor const&, bool) + 0x4b (0x7fbc0272115b in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#6:
#7: at::native::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) + 0x352 (0x7fbcab838312 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)
#8:
#9:
#10: at::Tensor::cpu() const + 0xda (0x7fbc02cb0eda in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#11: torch_mlu::OpMethods::fill_(at::Tensor&, c10::Scalar) + 0x43 (0x7fbc02c87bb3 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#12: torch_mlu::AtenMluType::fill_(at::Tensor&, c10::Scalar) + 0x4e (0x7fbc0272294e in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)
#13: c10::detail::wrap_kernel_functor_unboxed_<c10::detail::WrapKernelFunction_<at::Tensor& (at::Tensor&, c10::Scalar), &torch_mlu::AtenMluType::fill_, at::Tensor&, c10::guts::typelist::typelist
#14:
#15:
#16: at::Tensor::fill_(c10::Scalar) const + 0x16f (0x7fbcab56c4bf in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)
#17: at::native::full(c10::ArrayRef
#18: at::native::ones(c10::ArrayRef
#19:
#20:
#21: dcn_v2_cpu_forward(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, int, int, int, int, int, int, int, int, int) + 0x2c7 (0x7fbb6f0a3937 in /workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/_ext.cpython-37m-x86_64-linux-gnu.so)
#22: dcn_v2_forward(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, int, int, int, int, int, int, int, int, int) + 0x94 (0x7fbb6f092374 in /workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/_ext.cpython-37m-x86_64-linux-gnu.so)
#23:
#24:
#25:
#28: python() [0x4245b8]
#30: THPFunction_apply(_ *, _ *) + 0x983 (0x7fbd52017a23 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
#33: python() [0x4245b8]
#41: python() [0x497681]
#44: python() [0x4245b8]
#52: python() [0x497681]
#55: python() [0x4245b8]
#63: python() [0x497681]
热门帖子
精华帖子