You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/08/12 12:04:09 UTC
[GitHub] [incubator-mxnet] marekjg opened a new issue #15862: CPU GEMM in
float 16 fails silently with NaiveEngine (CPP api)
marekjg opened a new issue #15862: CPU GEMM in float 16 fails silently with NaiveEngine (CPP api)
URL: https://github.com/apache/incubator-mxnet/issues/15862
## Description
When running inference in cpp api with naive engine, the computation is not performed but the output array is initialized and no exception is raised. Without naive engine, there's exception thrown (which is expected). Using python api with naive engine will raise exception in both cases so the problem seems to be related to CPP api. It applies to Dense layer but also happend with model with RNN layers so my guess is that whenever GEMM is used, it fails.
## Environment info (Required)
```
----------Python Info----------
('Version :', '2.7.12')
('Compiler :', 'GCC 5.4.0 20160609')
('Build :', ('default', 'Nov 12 2018 14:36:49'))
('Arch :', ('64bit', 'ELF'))
------------Pip Info-----------
('Version :', '8.1.1')
('Directory :', '/usr/lib/python2.7/dist-packages/pip')
----------MXNet Info-----------
No MXNet installed.
----------System Info----------
('Platform :', 'Linux-4.4.0-157-generic-x86_64-with-Ubuntu-16.04-xenial')
('system :', 'Linux')
('node :', 'devdesktop')
('release :', '4.4.0-157-generic')
('version :', '#185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019')
----------Hardware Info----------
('machine :', 'x86_64')
('processor :', 'x86_64')
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 94
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Stepping: 3
CPU MHz: 3700.820
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
BogoMIPS: 6816.62
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0002 sec, LOAD: 0.9854 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0610 sec, LOAD: 1.2456 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0714 sec, LOAD: 0.5382 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0606 sec, LOAD: 0.4074 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0909 sec, LOAD: 1.0720 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0736 sec, LOAD: 0.9317 sec.
```
## Build info (Required if built from source)
gcc
MXNet commit hash:
57927a932610581a351e4aed4fb7c9209d0e57c2
Build config:
cmake -DUSE_CUDA=OFF -DUSE_CPP_PACKAGE=1 -GNinja .. && ninja
## Error Message:
With NaiveEngine:
```
[13:44:58] ../src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.3.1. Attempting to upgrade...
[13:44:58] ../src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
[13:44:58] ../src/engine/engine.cc:55: MXNet start using engine: NaiveEngine
```
Without NaiveEngine:
```
[13:44:54] ../src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.3.1. Attempting to upgrade...
[13:44:54] ../src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded!
terminate called after throwing an instance of 'dmlc::Error'
what(): [13:44:54] ./mxnet-cpp/ndarray.hpp:242: Check failed: MXNDArrayWaitAll() == 0 (-1 vs. 0) : [13:44:54] ../src/operator/nn/./fully_connected-inl.h:212: float16 fully connected layer is currentlyonly supported by CuDNN version.
Stack trace:
[bt] (0) ./test_cpp(dmlc::LogMessageFatal::~LogMessageFatal()+0x34) [0x407118]
[bt] (1) libmxnet.so(void mxnet::op::FullyConnectedCompute<mshadow::cpu>(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x218) [0x7fed1cd276e8]
[bt] (2) libmxnet.so(mxnet::op::FullyConnectedComputeExCPU(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&)+0x854) [0x7fed1cd1ee34]
[bt] (3) libmxnet.so(+0x83395d) [0x7fed1c73395d]
[bt] (4) libmxnet.so(+0x833a0f) [0x7fed1c733a0f]
[bt] (5) libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x500) [0x7fed1c7169e0]
[bt] (6) libmxnet.so(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#1}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&)+0xfe) [0x7fed1c716ffe]
[bt] (7) libmxnet.so(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)> (std::shared_ptr<dmlc::ManualEvent>)> >::_M_run()+0x4a) [0x7fed1c7156aa]
[bt] (8) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7fed1bc36c80]
Stack trace:
[bt] (0) ./test_cpp(dmlc::LogMessageFatal::~LogMessageFatal()+0x34) [0x407118]
[bt] (1) ./test_cpp() [0x40bdb0]
[bt] (2) ./test_cpp() [0x40599f]
[bt] (3) ./test_cpp() [0x405f86]
[bt] (4) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fed1b2b5830]
[bt] (5) ./test_cpp() [0x405119]
```
## Minimum reproducible example and steps to reproduce
Create the model on the machine with GPU (linear-symbol.json and linear-0000.params files) with the following script:
```
import mxnet as mx
from mxnet import nd
from mxnet.gluon import nn
ctx = mx.gpu() # fails when mx.cpu()
linear = nn.Dense(128)
linear.initialize(ctx=ctx)
linear.cast('float16')
x = nd.random_normal(shape=(1, 256), ctx=ctx, dtype='float16')
linear(x)
linear.export('linear')
```
run the inference with the following cpp program:
```
#include <mxnet-cpp/MxNetCpp.h>
#include <map>
using namespace mxnet::cpp;
using namespace std;
void set_input(NDArray &a)
{
vector<float> data(a.Size());
for(int i = 0; i < data.size(); ++i) {
data[i] = rand() / float(RAND_MAX);
}
a.SyncCopyFromCPU(data.data(), data.size());
}
void forward_pass(Executor *e)
{
// Set input
set_input(e->arg_dict()["data"]);
NDArray::WaitAll();
// Forward
e->Forward(false);
NDArray::WaitAll();
// Get output
vector<float> destination(e->outputs[0].Size());
e->outputs[0].SyncCopyToCPU(destination.data(), destination.size());
NDArray::WaitAll();
}
int main()
{
auto ctx = Context::cpu();
auto net = Symbol::Load("linear-symbol.json");
std::map<std::string, NDArray> args_map, aux_map;
std::map<std::string, NDArray> parameters;
NDArray::Load("linear-0000.params", 0, ¶meters);
for (const auto &k : parameters) {
if (k.first.substr(0, 4) == "aux:") {
auto name = k.first.substr(4, k.first.size() - 4);
aux_map[name] = k.second.Copy(ctx);
}
else if (k.first.substr(0, 4) == "arg:") {
auto name = k.first.substr(4, k.first.size() - 4);
args_map[name] = k.second.Copy(ctx);
}
}
Shape data_shape(1, 256);
NDArray data(data_shape, ctx, false, 2);
args_map["data"] = data;
NDArray::WaitAll();
Executor *e = net.SimpleBind(ctx, args_map);
forward_pass(e);
return 0;
}
```
## Steps to reproduce
1. Create the model as described above
2. Compile cpp program with gcc
3. Run the cpp program with NaiveEngine: `MXNET_ENGINE_TYPE="NaiveEngine" ./test` and observe no error
4. Run the cpp program with default engine: `./test` and observe the error with message that fp16 fully connected layer is not supported in CPU
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services