You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/06/18 21:14:36 UTC
[GitHub] [incubator-mxnet] jfrank94 opened a new issue #20363: Issues with GluonTS library (MXNet Error)
jfrank94 opened a new issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363
I'm running into an error when running the DeepAREstimator from the GluonTS library. I have downloaded all of the necessary packages, and with using the mx.gpu() function, it recognizes that the GPU (Nvidia Cuda 11.2) exists on the system
Note, that I'm able to run the code fine on Colab without having to run specific commands like installing "libquadmath0" or the NCCL library (ver 2.8.4 for cuda 11.2), but when running on a docker image, this seems to be the case.
Here's the full error trace:
`0%| | 0/1 [00:00<?, ?it/s]
learning rate from "lr_scheduler" has been overwritten by "learning_rate" in optimizer.
0%| | 0/1 [00:03<?, ?it/s]
---------------------------------------------------------------------------
MXNetError Traceback (most recent call last)
<ipython-input-7-33795d1460ee> in <module>
29 print("\nPatient {} - Amount of Days (Train): {}\n | Amount of Days (Valid): {}\n".format(p_id, train_days, valid_days))
30
---> 31 train1_output = estimator.train(training_data=training_data, validation_data=validation_data)
32 #print(agg_metrics)
/usr/local/lib/python3.6/dist-packages/gluonts/mx/model/estimator.py in train(self, training_data, validation_data, num_workers, num_prefetch, shuffle_buffer_length, cache_data, **kwargs)
205 num_prefetch=num_prefetch,
206 shuffle_buffer_length=shuffle_buffer_length,
--> 207 cache_data=cache_data,
208 ).predictor
/usr/local/lib/python3.6/dist-packages/gluonts/mx/model/estimator.py in train_model(self, training_data, validation_data, num_workers, num_prefetch, shuffle_buffer_length, cache_data)
177 net=training_network,
178 train_iter=training_data_loader,
--> 179 validation_iter=validation_data_loader,
180 )
181
/usr/local/lib/python3.6/dist-packages/gluonts/mx/trainer/_base.py in __call__(self, net, train_iter, validation_iter)
377 epoch_no,
378 train_iter,
--> 379 num_batches_to_use=self.num_batches_per_epoch,
380 )
381 if is_validation_available:
/usr/local/lib/python3.6/dist-packages/gluonts/mx/trainer/_base.py in loop(epoch_no, batch_iter, num_batches_to_use, is_training)
308 batch_size = loss.shape[0]
309
--> 310 if not np.isfinite(ndarray.sum(loss).asscalar()):
311 logger.warning(
312 "Batch [%d] of Epoch[%d] gave NaN loss and it will be ignored",
/usr/local/lib/python3.6/dist-packages/mxnet/ndarray/ndarray.py in asscalar(self)
2583 raise ValueError("The current array is not a scalar")
2584 if self.ndim == 1:
-> 2585 return self.asnumpy()[0]
2586 else:
2587 return self.asnumpy()[()]
/usr/local/lib/python3.6/dist-packages/mxnet/ndarray/ndarray.py in asnumpy(self)
2564 self.handle,
2565 data.ctypes.data_as(ctypes.c_void_p),
-> 2566 ctypes.c_size_t(data.size)))
2567 return data
2568
/usr/local/lib/python3.6/dist-packages/mxnet/base.py in check_call(ret)
244 """
245 if ret != 0:
--> 246 raise get_last_ffi_error()
247
248
MXNetError: Traceback (most recent call last):
File "../include/mshadow/././././cuda/tensor_gpu-inl.cuh", line 129
Name: Check failed: err == cudaSuccess (209 vs. 0) : MapPlanKernel ErrStr:no kernel image is available for execution on the device`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
szha commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-865093054
cc @lostella
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] github-actions[bot] commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-863580690
Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on [contributing to MXNet](https://mxnet.apache.org/community/contribute) and our [development guides wiki](https://cwiki.apache.org/confluence/display/MXNET/Developments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] jfrank94 commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
jfrank94 commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-864287772
@TristonC Maybe it might help to have the folks who worked on the GluonTS library help as well? Thanks for your help so far, btw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] jfrank94 commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
jfrank94 commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-866078003
@szha Alright, no problem. Thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] TristonC commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
TristonC commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-863617726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
szha commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-866073883
GluonTS support should go in awslabs/gluon-ts#1571 as mxnet doesn't have knowledge of the downstream application. If there's any unexpected behavior in mxnet, please share how that's reproduced in a bug report here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] TristonC edited a comment on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
TristonC edited a comment on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-864163997
Yeah, I could not reproduce the error in my local container (NGC container with MXNet 1.9) with GPU. I don't have access to Azure. @leezu Could you comment on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] jfrank94 closed issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
jfrank94 closed issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
szha commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-865093054
cc @lostella
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] lostella commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
lostella commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-865094784
looks like the same as https://github.com/awslabs/gluon-ts/issues/1571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] jfrank94 commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
jfrank94 commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-863621233
@TristonC The docker image version is 19.3.12, but it's running through a Kubernetes cluster on Azure. In order to reproduce this error, you should download these packages first on a Jupyter notebook:
!sudo apt install libquadmath0
!sudo apt install libnccl2=2.8.4-1+cuda11.2 --allow-downgrades -y
!pip install mxnet-cu112
!pip install --upgrade gluonts
!pip install scipy
!pip install xlrd==1.2.0
Then import the mxnet library as well as these libraries under GluonTS:
from gluonts.dataset.common import ListDataset
from gluonts.model.deepar import DeepAREstimator
from gluonts.mx.trainer import Trainer
from gluonts.dataset.util import to_pandas
from gluonts.evaluation import Evaluator
from gluonts.evaluation.backtest import make_evaluation_predictions
from gluonts.support.util import get_download_path
from gluonts.mx.trainer import Trainer
from gluonts.model.predictor import Predictor
Then, you can import a dataset and convert it into a ListDataSet like so:
training_data = ListDataset(
[{"start": train.index[0], "target": train.value}],
freq = "1H"
)
Finally, initialize the DeepAREstimator and train it like so:
estimator = DeepAREstimator(freq="1D", prediction_length=28, trainer=Trainer(epochs=60, ctx=ctx,
num_batches_per_epoch=1), num_parallel_samples=1)
train1_output = estimator.train(training_data=training_data, validation_data=validation_data)
I hope this helps!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] jfrank94 commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
jfrank94 commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-866069075
@TristonC @szha Are there any updates as of yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] TristonC commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
TristonC commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-864331276
The trace goes down to MXNet. But sure, @szha Any GluonTS friend can help on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] jfrank94 edited a comment on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
jfrank94 edited a comment on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-863621233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] lostella commented on issue #20363: Issues with GluonTS library (MXNet Error)
Posted by GitBox <gi...@apache.org>.
lostella commented on issue #20363:
URL: https://github.com/apache/incubator-mxnet/issues/20363#issuecomment-865094784
looks like the same as https://github.com/awslabs/gluon-ts/issues/1571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org