×
分享到微信

打开微信,使用扫一扫进入页面后,点击右上角菜单,

点击“发送给朋友”或“分享到朋友圈”完成分享

fill_ Op cannot run on MLU device, start running on CPU! yu_sjtu2021-08-31 19:36:54 回复 1 查看 技术答疑 使用求助
fill_ Op cannot run on MLU device, start running on CPU!
分享到:

大家好,当我在MLU模式下运行量化后的模型推理,在output = self.model(images)[-1]句遇到:

fill_ Op cannot run on MLU device, start running on CPU!

copy_ Op cannot run on MLU device, start running on CPU!

警告并报错,请问这是否代表该操作MLU算子不支持?还是有数据未迁移至MLU?

谢谢!


警告以及报错的文字如下:

[WARNING][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/operators/op_methods.cpp][line:637][fill_][thread:140451381954304][process:84710]:

fill_ Op cannot run on MLU device, start running on CPU!

[ERROR][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/core/tensor_impl.cpp][line:866][cpu_data][thread:140451381954304][process:84710]:


Both cpu_storage and mlu_storage are not initialized!

Please check is there any invalid tensor operates such as:

output = input.cpu() or output = input.to("cpu") in pytorch model when doing mlu/mfus inference.


Can not call cpu_data on an empty tensor.

[WARNING][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/operators/op_methods.cpp][line:68][copy_][thread:140451381954304][process:84710]:

copy_ Op cannot run on MLU device, start running on CPU!

[ERROR][/workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/core/tensor_impl.cpp][line:866][cpu_data][thread:140451381954304][process:84710]:


Both cpu_storage and mlu_storage are not initialized!

Please check is there any invalid tensor operates such as:

output = input.cpu() or output = input.to("cpu") in pytorch model when doing mlu/mfus inference.


Traceback (most recent call last):

  File "src/test.py", line 145, in <module>

    prefetch_test(opt)

  File "src/test.py", line 80, in prefetch_test

    ret = detector.run(pre_processed_images)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/detectors/ _detector.py", line 138, in run

    output, dets, forward_time = self.process(images, return_time=True)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/detectors/ctdet.py", line 30, in process

    output = self.model(images)[-1]

  File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__

    result = self.forward(*input, **kwargs)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 473, in forward

    x = self.dla_up(x)

  File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__

    result = self.forward(*input, **kwargs)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 412, in forward

    ida( s, len( s) -i - 2, len( s))

  File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__

    result = self.forward(*input, **kwargs)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 385, in forward

    s[i] = upsample(project( s[i]))

  File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__

    result = self.forward(*input, **kwargs)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/pose_dla_dcn.py", line 356, in forward

    x = self.conv(x)

  File "/workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__

    result = self.forward(*input, **kwargs)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/dcn_v2.py", line 128, in forward

    self.deformable_groups)

  File "/workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/dcn_v2.py", line 31, in forward

    ctx.deformable_groups)

RuntimeError: Can not call cpu_data on an empty tensor. (cpu_data at /workspace/volume/private/sdk/cambricon_pytorch/pytorch/src/catch/torch_mlu/csrc/aten/core/tensor_impl.cpp:870)

#0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x6a (0x7fbd5179416a in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libc10.so)

#1: torch_mlu::MLUTensorImpl::cpu_data() + 0xef4 (0x7fbc027589b4 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#2: torch_mlu::copy_to_cpu(at::Tensor&, at::Tensor const&) + 0xd6 (0x7fbc027683f6 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#3: torch_mlu::OpMethods::copy_(at::Tensor&, at::Tensor const&, bool) + 0x5f (0x7fbc02c6280f in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#4: torch_mlu::CnmlOps::copy_(at::Tensor&, at::Tensor const&, bool) + 0x2dc (0x7fbc02a0b60c in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#5: torch_mlu::AtenMluType::copy_(at::Tensor&, at::Tensor const&, bool) + 0x4b (0x7fbc0272115b in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#6: <unknown function> + 0xb62d36 (0x7fbcab836d36 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#7: at::native::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) + 0x352 (0x7fbcab838312 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#8: <unknown function> + 0xef52f6 (0x7fbcabbc92f6 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#9: <unknown function> + 0x5daf53 (0x7fbc02c61f53 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#10: at::Tensor::cpu() const + 0xda (0x7fbc02cb0eda in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#11: torch_mlu::OpMethods::fill_(at::Tensor&, c10::Scalar) + 0x43 (0x7fbc02c87bb3 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#12: torch_mlu::AtenMluType::fill_(at::Tensor&, c10::Scalar) + 0x4e (0x7fbc0272294e in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libaten_mlu.so)

#13: c10::detail::wrap_kernel_functor_unboxed_<c10::detail::WrapKernelFunction_<at::Tensor& (at::Tensor&, c10::Scalar), &torch_mlu::AtenMluType::fill_, at::Tensor&, c10::guts::typelist::typelist<at::Tensor&, c10::Scalar> >, at::Tensor& (at::Tensor&, c10::Scalar)>::call(c10::OperatorKernel*, at::Tensor&, c10::Scalar) + 0x3d (0x7fbc071f682d in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch_mlu/csrc/lib/libcatch_python.so)

#14: <unknown function> + 0x2c201e8 (0x7fbcad8f41e8 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#15: <unknown function> + 0xde8f0f (0x7fbcababcf0f in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#16: at::Tensor::fill_(c10::Scalar) const + 0x16f (0x7fbcab56c4bf in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#17: at::native::full(c10::ArrayRef<long>, c10::Scalar, c10::TensorOptions const&) + 0x6b (0x7fbcab843deb in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#18: at::native::ones(c10::ArrayRef<long>, c10::TensorOptions const&) + 0x38 (0x7fbcab844168 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#19: <unknown function> + 0xf60975 (0x7fbcabc34975 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#20: <unknown function> + 0x2a9b135 (0x7fbcad76f135 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch.so)

#21: dcn_v2_cpu_forward(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, int, int, int, int, int, int, int, int, int) + 0x2c7 (0x7fbb6f0a3937 in /workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/_ext.cpython-37m-x86_64-linux-gnu.so)

#22: dcn_v2_forward(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, int, int, int, int, int, int, int, int, int) + 0x94 (0x7fbb6f092374 in /workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/_ext.cpython-37m-x86_64-linux-gnu.so)

#23: <unknown function> + 0x19223 (0x7fbb6f0a0223 in /workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/_ext.cpython-37m-x86_64-linux-gnu.so)

#24: <unknown function> + 0x1947e (0x7fbb6f0a047e in /workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/_ext.cpython-37m-x86_64-linux-gnu.so)

#25: <unknown function> + 0x1573c (0x7fbb6f09c73c in /workspace/volume/private/10_CenterNet_0.06/CenterNet/src/lib/models/networks/DCNv2/_ext.cpython-37m-x86_64-linux-gnu.so)

<omitting python s>

#28: python() [0x4245b8]

#30: THPFunction_apply(_ *, _ *) + 0x983 (0x7fbd52017a23 in /workspace/volume/private/sdk/venv/pytorch/lib/python3.7/site-packages/torch/lib/libtorch_python.so)

#33: python() [0x4245b8]

#41: python() [0x497681]

#44: python() [0x4245b8]

#52: python() [0x497681]

#55: python() [0x4245b8]

#63: python() [0x497681]


版权所有 © 2021 寒武纪 Cambricon.com 备案/许可证号:京ICP备17003415号-1
关闭