You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@mxnet.apache.org by Elias via MXNet Forum <mx...@discoursemail.com.INVALID> on 2021/07/07 08:33:14 UTC
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
I'm using the following code to create an array on gpu and the program never finishes. I got no error, the program just keeps running.
```
import mxnet as mx
mx.ndarray.ones(shape=(1,1),ctx=mx.gpu(0))
```
The code runs as expected both on my other GPU and on CPU. I've checked the GPU usage using nvidia-smi.exe and the memory usage is continuously increasing from a starting value of approximately 90MiB. nvidia-smi gives the following output
```
+-----------------------------------------------------------------------------+
Wed Jul 7 09:57:30 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 471.11 Driver Version: 471.11 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro RTX 4000 TCC | 00000000:65:00.0 Off | N/A |
| 30% 51C P8 19W / 125W | 306MiB / 8063MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Quadro P400 WDDM | 00000000:B3:00.0 On | N/A |
| 34% 38C P8 N/A / N/A | 216MiB / 2048MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2960 C ...iles\Python39\pythonw.exe 305MiB |
etc.
```
I'm using the following :
* Windows server 2019 Essentials
* Quadro RTX 4000
* NVIDIA Graphics driver 471.11
* Cuda 10.2.86
* Python 3.9.5
* pip 21.1.1
* mxnet-cu102 2.0.0b20201108
* cuDNN 6.14.11.10020
I've tested both TCC and WDDM modes on the GPU. I've also used driver version 441.66 where the cuda version displayed by nvidia-smi is 10.2 but the issue persists. I've also tested mxnet-cu102 1.7.0. Both mxnet versions were downloaded from [https://dist.mxnet.io/python/cu102](https://dist.mxnet.io/python/cu102) and installed using pip. I've tried reinstalling all the programs. I run the code using IDLE.
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/1) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/905f76722606a4acee6e5c5662cc4291880e30ad1a13886a3186f9a2e100b8a0).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by Elias via MXNet Forum <mx...@discoursemail.com.INVALID>.
I asked the same question [here](https://github.com/apache/incubator-mxnet/issues/20531) and found out that the array is created after roughly 15 min.
The problem can be fixed by creating an environment variable "CUDA_CACHE_MAXSIZE". 1 GiB was a good value, at least for me. After this, the 15 min wait only occurs the first time a GPU is used and the subsequent runs are a lot faster, even if python is closed in between.
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/9) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/7d6c5684eb309631b3feb58db299c14bc1bfe0630fa213af708e3409aa4e7c44).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by kim via MXNet Forum <mx...@discoursemail.com.INVALID>.
In addition to comparing the quotes of popular companies, it is also important to take a look at what different options are available. Some of the options that you may want to consider include [https://mlsdev.com](https://mlsdev.com) app, which allows users with Apple devices the ability to install their own version of apps from the Apple app store. On the other hand, there are also many other options available, such as DIY app, which is an option open to people who want to develop a mobile application for others without having to spend any money upfront. Some of the best known app developers in the field are Coding Cat, Appiction, and Rocket Application Development, all of which have received good feedback from the market.
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/8) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/5531d9bdcf6d75421062f1d3e97d6f107c17e8e6eb0e5f5135ba651f7ee83232).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by kim via MXNet Forum <mx...@discoursemail.com.INVALID>.
Thanks for the interesting solution!
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/7) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/d92489e5e94222b1e9aaadfdbb7567639cefe6016d0f427da132d31c25d7e68f).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by Elias via MXNet Forum <mx...@discoursemail.com.INVALID>.
I cannot find a 1.8 pip package for windows and I don't really know how to compile from source. I tried following the compiling instructions. I didn't succeed and I don't even know how to find out what I'm doing wrong or what is not working.
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/6) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/ce39eebbb6920b0e02a8c5aeb2df17ed864044ec59b265f8513e5962e99f14b1).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by Triston via MXNet Forum <mx...@discoursemail.com.INVALID>.
@Elias,
Would you mind giving MXNet 1.8 a try if it is possible?
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/5) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/f2ef5e12b1fab6c17cfa00e2e122d7885f4a729c97e1886812ebff2eaa60db62).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by Elias via MXNet Forum <mx...@discoursemail.com.INVALID>.
I tried using mxnet-cu92 1.5.0 and both GPUs work now. I'd however like using cuda 10.2 as I have other stuff that use it (and it bugs me that I don't know what I'm doing wrong).
I think the problem comes from something that happened between mxnet 1.5.0 and mxnet 1.6.0. I checked the release notes of 1.5.1 and 1.6.0. and the only change that seems relevant to me is https://github.com/apache/incubator-mxnet/pull/15449. Has something else changed in between that I should check?
I tried using mxnet 1.6.0 and downgrading to cuDNN v.7.6.5.32 (had v8.2.2.26 before) but it didn't help.
Also found a similar discussion https://discuss.mxnet.apache.org/t/using-mxnet-with-cuda-freezes-python/6952
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/4) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/e23abd1456411209470f3ef4fc700b870b057c477d6983dfca1188c4da8f0ab4).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by Elias via MXNet Forum <mx...@discoursemail.com.INVALID>.
Is there a pip package somewhere for mxnet-cu11x for Windows? I was only able to find linux versions from PyPI and [https://dist.mxnet.io/python](https://dist.mxnet.io/python).
Meanwhile, I tested using cuda 9.2 and with both mxnet-cu92 1.6.0 and 1.7.0. The results were the same.
The output of nvidia-smi is incorrect as far as I know and is a result of updating the drivers. I haven't installed cuda 11 (and the folder for that version does not exist). I've ran the code using driver version 441.66, which shows the correct cuda version and the code does not work.
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/3) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/8cd98726485bf0e4ea603031d04ea38ec31adf3b71620b8f8f65e3aae9180ab3).
[MXNet Forum] [Discussion] Creating a NDArray on gpu does not finish
Posted by Triston via MXNet Forum <mx...@discoursemail.com.INVALID>.
@Elias The system has the latest CUDA. Could you try if you can install MXNet 1.8 with CUDA 11?
---
[Visit Topic](https://discuss.mxnet.apache.org/t/creating-a-ndarray-on-gpu-does-not-finish/6995/2) or reply to this email to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click here](https://discuss.mxnet.apache.org/email/unsubscribe/f0ade1aec456c144a5c62f0d3deb07343220d0a248daf52b794661a49a8131c0).