You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/01/22 14:01:10 UTC
[GitHub] [incubator-mxnet] NamelessLoop opened a new issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
NamelessLoop opened a new issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778
I have already "cythonized" my code declaring in the code int, double, list ecc. But, as my code has data type from the Mxnet library like: (mxnet.io.NDArrayIter, mxnet.ndarray.ndarray.NDArray), I wanted to declare those to speed up the training phase of the code. Searching on the web I find out that, like for numpy.get_include() in the setup.py file, I have to declare the same path for mxnet in "include_dirs=...". The command for mxnet is mxnet.libinfo.find_include_path(), but it gives me this error: `RuntimeError: Cannot find the MXNet include path in either C:\Users\pietr\Anaconda3\envs\analysis\lib\site-packages\mxnet\include/ or C:\Users\pietr\Anaconda3\envs\analysis\lib\site-packages\mxnet\../../include/
`
Therefore, I do not know how to write properly the setup.py file, furthermore what type should I put in for example:
```
cdef ???? encoder_input = trn_batch.data[0].as_in_context(mx.cpu())
cdef ???? train_iter = mxnet.io.NDArrayIter(train_input, train_output, batch_size=train_bs, shuffle=True)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765442484
Sorry, but what exactly are you trying to do? Do you want to directly interface with MXNet C API in your Cython code? Or do you want to use MXNet Python API in your Cython code? In the latter case, you can't release the GIL and the MXNet code will run in standard Python invoked by your Cython code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] github-actions[bot] commented on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765418700
Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on [contributing to MXNet](https://mxnet.apache.org/community/contribute) and our [development guides wiki](https://cwiki.apache.org/confluence/display/MXNET/Developments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop removed a comment on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop removed a comment on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-766094856
I found out this website https://mxnet.apache.org/versions/1.1.0/doxygen/files.html where this files should be the include folder I have to link to the setup.py file. Am I getting this wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop commented on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop commented on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765465584
The python_code.pyx:
`
import mxnet
cimport numpy
cimport mxnet???
def MxNetTrain(net, model_type, trainer, loss_function, train_input, train_output, val_input, val_output, test_input,
test_output, epochs, args_save):
# In case of attention training un-pack the Tuple objects
if (model_type == 'seq2seq - attention') or model_type == 'seq2seq - transformer':
encoder_rnn, attn_decoder = net[0], net[1]
trainer = trainer[1]
# init validation loss as infinity
cdef double best_val
best_val = float("Inf")
# initializing arrays to save parameters through epochs
cdef list learning_rates = []
cdef list validation_losses = []
cdef list test_losses = []
cdef list training_losses = []
with open(args_save, 'w') as fp: # condition to avoid Jankins/file_directory error
pass
cdef int train_bs = train_input.shape[0]
cdef int val_bs = val_input.shape[0]
cdef int test_bs = test_input.shape[0]
# creating nd.arrayIter (training, validation and testing)
train_iter = mx.io.NDArrayIter(train_input, train_output, batch_size=train_bs, shuffle=True)
val_iter = mx.io.NDArrayIter(val_input, val_output, batch_size=val_bs, shuffle=True)
test_iter = mx.io.NDArrayIter(test_input, test_output, batch_size=test_bs, shuffle=True)
cdef double total_loss = 0.0
cdef double ntotal_training = 0.0
cdef ??? encoder_input
cdef ??? decoder_target
# for epoch in tqdm_notebook(range(epochs), desc='epochs'):
iepoch = int()
for epoch in range(epochs):
train_iter.reset()
val_iter.reset()
test_iter.reset()
for trn_batch in train_iter:
if (model_type == 'seq2seq - attention') or (model_type == 'seq2seq - transformer'):
encoder_input = trn_batch.data[0].as_in_context(mx.cpu())
decoder_target = trn_batch.label[0].as_in_context(mx.cpu())`
Then the setup.py;
`from distutils.core import setup
import numpy
from Cython.Build import cythonize
setup(
ext_modules= cythonize("MxRnnBoosted.pyx"),
include_dirs=[numpy.get_include(), mxnet_get_include_path_]
)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop commented on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop commented on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-766094856
I found out this website https://mxnet.apache.org/versions/1.1.0/doxygen/files.html where this files should be the include folder I have to link to the setup.py file. Am I getting this wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop edited a comment on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop edited a comment on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765451715
Exactly I have a python code in which I'm using mxnet functions. Yesterday I found out that if I compile it, it is faster but moreover if I add extensions to variables in the .pyx and then compile it, it is even more faster. I got no problem with numpy.ndarray but I've no idea how to add data type to those mxnet arrays type and to declare the get_include path in the setup.py file to compile the .pyx because the function provided by the website is giving me Runtime error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop commented on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop commented on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765451715
Exactly I have a python code in which I'm using mxnet functions. Yesterday I found out that if I compile it, it is faster but moreover if I add extension to the .pyx and then compile it, it is even more faster. I got no problem with numpy.ndarray but I've no idea how to add extension to those mxnet arrays type and to declare the get_include path in the setup.py file to compile the .pyx because the function provided by the website is giving me Runtime error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765485551
It's possible that you need to modify the MXNet source to enable `cimport mxnet`. Your contribution to making this possible would be very welcome
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop edited a comment on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop edited a comment on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765451715
Exactly I have a python code in which I'm using mxnet functions. Yesterday I found out that if I compile it, it is faster but moreover if I add extensions to variables in the .pyx and then compile it, it is even more faster. I got no problem with numpy.ndarray but I've no idea how to add extension to those mxnet arrays type and to declare the get_include path in the setup.py file to compile the .pyx because the function provided by the website is giving me Runtime error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765442484
Sorry, but what exactly are you trying to do? Do you want to directly interface with MXNet C API in your Cython code? Or do you want to use MXNet Python API in your Cython code? I think in the latter case, you can't release the GIL and the MXNet code will run in standard Python invoked by your Cython code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop edited a comment on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop edited a comment on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765465584
The python_code.pyx:
```
import mxnet
cimport numpy
cimport mxnet???
def MxNetTrain(net, model_type, trainer, loss_function, train_input, train_output, val_input, val_output, test_input,
test_output, epochs, args_save):
# In case of attention training un-pack the Tuple objects
if (model_type == 'seq2seq - attention') or model_type == 'seq2seq - transformer':
encoder_rnn, attn_decoder = net[0], net[1]
trainer = trainer[1]
# init validation loss as infinity
cdef double best_val
best_val = float("Inf")
# initializing arrays to save parameters through epochs
cdef list learning_rates = []
cdef list validation_losses = []
cdef list test_losses = []
cdef list training_losses = []
with open(args_save, 'w') as fp: # condition to avoid Jankins/file_directory error
pass
cdef int train_bs = train_input.shape[0]
cdef int val_bs = val_input.shape[0]
cdef int test_bs = test_input.shape[0]
# creating nd.arrayIter (training, validation and testing)
train_iter = mx.io.NDArrayIter(train_input, train_output, batch_size=train_bs, shuffle=True)
val_iter = mx.io.NDArrayIter(val_input, val_output, batch_size=val_bs, shuffle=True)
test_iter = mx.io.NDArrayIter(test_input, test_output, batch_size=test_bs, shuffle=True)
cdef double total_loss = 0.0
cdef double ntotal_training = 0.0
cdef ??? encoder_input
cdef ??? decoder_target
# for epoch in tqdm_notebook(range(epochs), desc='epochs'):
iepoch = int()
for epoch in range(epochs):
train_iter.reset()
val_iter.reset()
test_iter.reset()
for trn_batch in train_iter:
if (model_type == 'seq2seq - attention') or (model_type == 'seq2seq - transformer'):
encoder_input = trn_batch.data[0].as_in_context(mx.cpu())
decoder_target = trn_batch.label[0].as_in_context(mx.cpu())`
```
Then the setup.py;
```
`from distutils.core import setup
import numpy
from Cython.Build import cythonize
setup(
ext_modules= cythonize("MxRnnBoosted.pyx"),
include_dirs=[numpy.get_include(), mxnet_get_include_path_]
)`
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] NamelessLoop commented on issue #19778: Cython setup.py including mxnet library, mxnet.libinfo.find_include_path() error
Posted by GitBox <gi...@apache.org>.
NamelessLoop commented on issue #19778:
URL: https://github.com/apache/incubator-mxnet/issues/19778#issuecomment-765493196
Therefore, there is no way to cdef an mx.io.NDArrayIter in a .pyx file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org