You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by "Sunderland, Kellen" <ke...@amazon.de> on 2017/09/26 14:42:30 UTC
CI system seems to be using python3 for python2 builds
I’ve been noticing in a few failed builds that the stack trace indicates we’re actually running python 3.4 in the python 2 tests. I know the CI folks are working hard getting everything setup, is this a known issue for the CI team?
For example: https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
Steps Python2: MKLML-CPU
StackTrace:
Stack trace returned 10 entries:
[bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
[bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b) [0x7fadba32c01b]
[bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vectorIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
[bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245) [0x7fadba18f165]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fadde26cadc]
[bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fadde26c40c]
[bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
[bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
[bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
[bt] (9) /usr/bin/python3() [0x48f2df]
-Kellen
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
Re: CI system seems to be using python3 for python2 builds
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Rahul,
Thanks for sharing the information and taking on the problem!
I will also try the option(using nosetests-2.7 in Jenkinsfile).
Regards,
- Tsuyoshi
On Thu, Sep 28, 2017 at 3:34 PM, Rahul Huilgol <ra...@gmail.com> wrote:
> Hi Gautam,
>
> I see that ‘nosetests’ is the command used to run python2 tests. It looks
> like that’s being mapped to use python3. I’ve checked that this is the case
> on my Ubuntu instance. I need to use ‘nosetests-2.7’ to use python2 for the
> tests. Please check if this fix works in the build environment
> (slave/docker container) as well.
>
> The PR you refer to only parallelized tests that were running one after the
> other, this command was being used even before that PR.
>
> Regards,
> Rahul
>
> On Wed, 27 Sep 2017 at 22:46 Gautam <ga...@gmail.com> wrote:
>
>> Hi Ozawa,
>>
>> Thanks for follow up.
>> Unfortunately I didn't get time to work on this today.
>>
>> However I have couple of points to mentions.
>> 1. Looks like this backtrace has been present since long time, since this
>> was not a test failure or build failure we never got notified about it.
>> Here
>> <
>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull
>> >
>> is the recent build log where back trace is present but build succeeded.
>>
>> 2. I don't think the default version of python on Ubuntu is 3.0, I logged
>> into one of the apache slave and the default version of Python is 2.7.6
>>
>> 3. There has been slight change
>> <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file
>> where
>> we tried to parallelize python2 and 3 test run. I am not sure if it
>> affects. I can probably scrub the build log and figure out if thats the
>> case.
>>
>>
>> Feel free to send the PR, if you have it ready.
>>
>>
>> -Gautam
>>
>>
>> On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>
>> > Hi Kumar,
>> >
>> > Thanks for looking into the issue. How is the progress of this problem?
>> > Shouldn't we call /usr/bin/env python2 or python2.7 in following
>> > source code instead of python since MXNet only supports python2
>> > currently?
>> > I think default version of python in Ubuntu is now python3, so it can
>> > cause the problem.
>> > If you have not yet done the work, I can create a PR for that in this
>> > weekend.
>> >
>> > ./python/mxnet/__init__.py:#!/usr/bin/env python
>> > ./python/mxnet/log.py:#!/usr/bin/env python
>> > ./tests/nightly/dist_lenet.py:#!/usr/bin/env python
>> > ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python
>> > ./tests/nightly/multi_lenet.py:#!/usr/bin/env python
>> > ./tests/nightly/test_kvstore.py:#!/usr/bin/env python
>> > ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python
>> > ./tools/ipynb2md.py:#!/usr/bin/env python
>> > ./tools/kill-mxnet.py:#!/usr/bin/env python
>> > ./tools/launch.py:#!/usr/bin/env python
>> > ./tools/parse_log.py:#!/usr/bin/env python
>> >
>> > On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <ke...@amazon.de>
>> > wrote:
>> > > Many thanks Gautam.
>> > >
>> > > On 9/26/17, 8:37 PM, "Kumar, Gautam" <ga...@amazon.com> wrote:
>> > >
>> > > Hi Kellen,
>> > >
>> > > This issue has been happening since last 3-4 days along with few
>> > other test failure.
>> > > I am looking into it.
>> > >
>> > > -Gautam
>> > >
>> > > On 9/26/17, 7:45 AM, "Sunderland, Kellen" <ke...@amazon.de>
>> wrote:
>> > >
>> > > I’ve been noticing in a few failed builds that the stack trace
>> > indicates we’re actually running python 3.4 in the python 2 tests. I know
>> > the CI folks are working hard getting everything setup, is this a known
>> > issue for the CI team?
>> > >
>> > > For example: https://builds.apache.org/
>> > blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
>> > >
>> > > Steps Python2: MKLML-CPU
>> > >
>> > > StackTrace:
>> > > Stack trace returned 10 entries:
>> > > [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> > ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
>> > > [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> > ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_
>> >
>> 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_
>> >
>> EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b)
>> > [0x7fadba32c01b]
>> > > [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> > ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto
>> > rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
>> > > [bt] (3)
>> /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245)
>> > [0x7fadba18f165]
>> > > [bt] (4)
>> /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c)
>> > [0x7fadde26cadc]
>> > > [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc)
>> > [0x7fadde26c40c]
>> > > [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
>> > x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
>> > > [bt] (7) /usr/lib/p
>> <https://maps.google.com/?q=(7)+/usr/lib/p&entry=gmail&source=g>
>> ython3.4/lib-dynload/_ctypes.cpython-34m-
>> > x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
>> > > [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
>> > > [bt] (9) /usr/bin/python3() [0x48f2df]
>> > >
>> > > -Kellen
>> > > Amazon Development Center Germany GmbH
>> > > Berlin - Dresden - Aachen
>> > > main office: Krausenstr. 38, 10117 Berlin
>> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> > > Ust-ID: DE289237879
>> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>> > >
>> > >
>> > >
>> > >
>> > > Amazon Development Center Germany GmbH
>> > > Berlin - Dresden - Aachen
>> > > main office: Krausenstr. 38, 10117 Berlin
>> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> > > Ust-ID: DE289237879
>> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>> >
>>
>>
>>
>> --
>> Best Regards,
>> Gautam Kumar
>>
Re: CI system seems to be using python3 for python2 builds
Posted by Rahul Huilgol <ra...@gmail.com>.
Hi Gautam,
I see that ‘nosetests’ is the command used to run python2 tests. It looks
like that’s being mapped to use python3. I’ve checked that this is the case
on my Ubuntu instance. I need to use ‘nosetests-2.7’ to use python2 for the
tests. Please check if this fix works in the build environment
(slave/docker container) as well.
The PR you refer to only parallelized tests that were running one after the
other, this command was being used even before that PR.
Regards,
Rahul
On Wed, 27 Sep 2017 at 22:46 Gautam <ga...@gmail.com> wrote:
> Hi Ozawa,
>
> Thanks for follow up.
> Unfortunately I didn't get time to work on this today.
>
> However I have couple of points to mentions.
> 1. Looks like this backtrace has been present since long time, since this
> was not a test failure or build failure we never got notified about it.
> Here
> <
> https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull
> >
> is the recent build log where back trace is present but build succeeded.
>
> 2. I don't think the default version of python on Ubuntu is 3.0, I logged
> into one of the apache slave and the default version of Python is 2.7.6
>
> 3. There has been slight change
> <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file
> where
> we tried to parallelize python2 and 3 test run. I am not sure if it
> affects. I can probably scrub the build log and figure out if thats the
> case.
>
>
> Feel free to send the PR, if you have it ready.
>
>
> -Gautam
>
>
> On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> > Hi Kumar,
> >
> > Thanks for looking into the issue. How is the progress of this problem?
> > Shouldn't we call /usr/bin/env python2 or python2.7 in following
> > source code instead of python since MXNet only supports python2
> > currently?
> > I think default version of python in Ubuntu is now python3, so it can
> > cause the problem.
> > If you have not yet done the work, I can create a PR for that in this
> > weekend.
> >
> > ./python/mxnet/__init__.py:#!/usr/bin/env python
> > ./python/mxnet/log.py:#!/usr/bin/env python
> > ./tests/nightly/dist_lenet.py:#!/usr/bin/env python
> > ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python
> > ./tests/nightly/multi_lenet.py:#!/usr/bin/env python
> > ./tests/nightly/test_kvstore.py:#!/usr/bin/env python
> > ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python
> > ./tools/ipynb2md.py:#!/usr/bin/env python
> > ./tools/kill-mxnet.py:#!/usr/bin/env python
> > ./tools/launch.py:#!/usr/bin/env python
> > ./tools/parse_log.py:#!/usr/bin/env python
> >
> > On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <ke...@amazon.de>
> > wrote:
> > > Many thanks Gautam.
> > >
> > > On 9/26/17, 8:37 PM, "Kumar, Gautam" <ga...@amazon.com> wrote:
> > >
> > > Hi Kellen,
> > >
> > > This issue has been happening since last 3-4 days along with few
> > other test failure.
> > > I am looking into it.
> > >
> > > -Gautam
> > >
> > > On 9/26/17, 7:45 AM, "Sunderland, Kellen" <ke...@amazon.de>
> wrote:
> > >
> > > I’ve been noticing in a few failed builds that the stack trace
> > indicates we’re actually running python 3.4 in the python 2 tests. I know
> > the CI folks are working hard getting everything setup, is this a known
> > issue for the CI team?
> > >
> > > For example: https://builds.apache.org/
> > blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
> > >
> > > Steps Python2: MKLML-CPU
> > >
> > > StackTrace:
> > > Stack trace returned 10 entries:
> > > [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_
> > ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
> > > [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_
> > ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_
> >
> 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_
> >
> EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b)
> > [0x7fadba32c01b]
> > > [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_
> > ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto
> > rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
> > > [bt] (3)
> /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245)
> > [0x7fadba18f165]
> > > [bt] (4)
> /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c)
> > [0x7fadde26cadc]
> > > [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc)
> > [0x7fadde26c40c]
> > > [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
> > x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
> > > [bt] (7) /usr/lib/p
> <https://maps.google.com/?q=(7)+/usr/lib/p&entry=gmail&source=g>
> ython3.4/lib-dynload/_ctypes.cpython-34m-
> > x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
> > > [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
> > > [bt] (9) /usr/bin/python3() [0x48f2df]
> > >
> > > -Kellen
> > > Amazon Development Center Germany GmbH
> > > Berlin - Dresden - Aachen
> > > main office: Krausenstr. 38, 10117 Berlin
> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > Ust-ID: DE289237879
> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> > >
> > >
> > >
> > >
> > > Amazon Development Center Germany GmbH
> > > Berlin - Dresden - Aachen
> > > main office: Krausenstr. 38, 10117 Berlin
> > > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > > Ust-ID: DE289237879
> > > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> >
>
>
>
> --
> Best Regards,
> Gautam Kumar
>
Re: CI system seems to be using python3 for python2 builds
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Thanks for your sharing information.
> 1. Looks like this backtrace has been present since long time, since this
> was not a test failure or build failure we never got notified about it. Here
> <https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull>
> is the recent build log where back trace is present but build succeeded.
Oh, my... but it's very essential information. I need more survey.
To understand what is real cause of the problem(on program-side or
infra-side), I would like to run Jenkinsfile locally.
Please let me give a time for a while to check - including whether
#7658 is real cause(it looks not).
> 2. I don't think the default version of python on Ubuntu is 3.0, I logged
> into one of the apache slave and the default version of Python is 2.7.6
Ouch, the "python" command is an alias of python2.7 on Ubuntu as you
mentioned. Thanks for your pointing out.
I will share information here when I have a progress, so I appreciate
if you do so too.
Thanks,
- Tsuyoshi
On Thu, Sep 28, 2017 at 2:46 PM, Gautam <ga...@gmail.com> wrote:
> Hi Ozawa,
>
> Thanks for follow up.
> Unfortunately I didn't get time to work on this today.
>
> However I have couple of points to mentions.
> 1. Looks like this backtrace has been present since long time, since this
> was not a test failure or build failure we never got notified about it. Here
> <https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull>
> is the recent build log where back trace is present but build succeeded.
>
> 2. I don't think the default version of python on Ubuntu is 3.0, I logged
> into one of the apache slave and the default version of Python is 2.7.6
>
> 3. There has been slight change
> <https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file where
> we tried to parallelize python2 and 3 test run. I am not sure if it
> affects. I can probably scrub the build log and figure out if thats the
> case.
>
>
> Feel free to send the PR, if you have it ready.
>
>
> -Gautam
>
>
> On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
>> Hi Kumar,
>>
>> Thanks for looking into the issue. How is the progress of this problem?
>> Shouldn't we call /usr/bin/env python2 or python2.7 in following
>> source code instead of python since MXNet only supports python2
>> currently?
>> I think default version of python in Ubuntu is now python3, so it can
>> cause the problem.
>> If you have not yet done the work, I can create a PR for that in this
>> weekend.
>>
>> ./python/mxnet/__init__.py:#!/usr/bin/env python
>> ./python/mxnet/log.py:#!/usr/bin/env python
>> ./tests/nightly/dist_lenet.py:#!/usr/bin/env python
>> ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python
>> ./tests/nightly/multi_lenet.py:#!/usr/bin/env python
>> ./tests/nightly/test_kvstore.py:#!/usr/bin/env python
>> ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python
>> ./tools/ipynb2md.py:#!/usr/bin/env python
>> ./tools/kill-mxnet.py:#!/usr/bin/env python
>> ./tools/launch.py:#!/usr/bin/env python
>> ./tools/parse_log.py:#!/usr/bin/env python
>>
>> On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <ke...@amazon.de>
>> wrote:
>> > Many thanks Gautam.
>> >
>> > On 9/26/17, 8:37 PM, "Kumar, Gautam" <ga...@amazon.com> wrote:
>> >
>> > Hi Kellen,
>> >
>> > This issue has been happening since last 3-4 days along with few
>> other test failure.
>> > I am looking into it.
>> >
>> > -Gautam
>> >
>> > On 9/26/17, 7:45 AM, "Sunderland, Kellen" <ke...@amazon.de> wrote:
>> >
>> > I’ve been noticing in a few failed builds that the stack trace
>> indicates we’re actually running python 3.4 in the python 2 tests. I know
>> the CI folks are working hard getting everything setup, is this a known
>> issue for the CI team?
>> >
>> > For example: https://builds.apache.org/
>> blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
>> >
>> > Steps Python2: MKLML-CPU
>> >
>> > StackTrace:
>> > Stack trace returned 10 entries:
>> > [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
>> > [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_
>> 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_
>> EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b)
>> [0x7fadba32c01b]
>> > [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_
>> ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto
>> rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
>> > [bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245)
>> [0x7fadba18f165]
>> > [bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c)
>> [0x7fadde26cadc]
>> > [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc)
>> [0x7fadde26c40c]
>> > [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
>> x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
>> > [bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
>> x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
>> > [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
>> > [bt] (9) /usr/bin/python3() [0x48f2df]
>> >
>> > -Kellen
>> > Amazon Development Center Germany GmbH
>> > Berlin - Dresden - Aachen
>> > main office: Krausenstr. 38, 10117 Berlin
>> > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> > Ust-ID: DE289237879
>> > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>> >
>> >
>> >
>> >
>> > Amazon Development Center Germany GmbH
>> > Berlin - Dresden - Aachen
>> > main office: Krausenstr. 38, 10117 Berlin
>> > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
>> > Ust-ID: DE289237879
>> > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>>
>
>
>
> --
> Best Regards,
> Gautam Kumar
Re: CI system seems to be using python3 for python2 builds
Posted by Gautam <ga...@gmail.com>.
Hi Ozawa,
Thanks for follow up.
Unfortunately I didn't get time to work on this today.
However I have couple of points to mentions.
1. Looks like this backtrace has been present since long time, since this
was not a test failure or build failure we never got notified about it. Here
<https://builds.apache.org/view/Incubator%20Projects/job/incubator-mxnet/job/master/448/consoleFull>
is the recent build log where back trace is present but build succeeded.
2. I don't think the default version of python on Ubuntu is 3.0, I logged
into one of the apache slave and the default version of Python is 2.7.6
3. There has been slight change
<https://github.com/apache/incubator-mxnet/pull/7658> in Jenkins file where
we tried to parallelize python2 and 3 test run. I am not sure if it
affects. I can probably scrub the build log and figure out if thats the
case.
Feel free to send the PR, if you have it ready.
-Gautam
On Wed, Sep 27, 2017 at 9:39 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi Kumar,
>
> Thanks for looking into the issue. How is the progress of this problem?
> Shouldn't we call /usr/bin/env python2 or python2.7 in following
> source code instead of python since MXNet only supports python2
> currently?
> I think default version of python in Ubuntu is now python3, so it can
> cause the problem.
> If you have not yet done the work, I can create a PR for that in this
> weekend.
>
> ./python/mxnet/__init__.py:#!/usr/bin/env python
> ./python/mxnet/log.py:#!/usr/bin/env python
> ./tests/nightly/dist_lenet.py:#!/usr/bin/env python
> ./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python
> ./tests/nightly/multi_lenet.py:#!/usr/bin/env python
> ./tests/nightly/test_kvstore.py:#!/usr/bin/env python
> ./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python
> ./tools/ipynb2md.py:#!/usr/bin/env python
> ./tools/kill-mxnet.py:#!/usr/bin/env python
> ./tools/launch.py:#!/usr/bin/env python
> ./tools/parse_log.py:#!/usr/bin/env python
>
> On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <ke...@amazon.de>
> wrote:
> > Many thanks Gautam.
> >
> > On 9/26/17, 8:37 PM, "Kumar, Gautam" <ga...@amazon.com> wrote:
> >
> > Hi Kellen,
> >
> > This issue has been happening since last 3-4 days along with few
> other test failure.
> > I am looking into it.
> >
> > -Gautam
> >
> > On 9/26/17, 7:45 AM, "Sunderland, Kellen" <ke...@amazon.de> wrote:
> >
> > I’ve been noticing in a few failed builds that the stack trace
> indicates we’re actually running python 3.4 in the python 2 tests. I know
> the CI folks are working hard getting everything setup, is this a known
> issue for the CI team?
> >
> > For example: https://builds.apache.org/
> blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
> >
> > Steps Python2: MKLML-CPU
> >
> > StackTrace:
> > Stack trace returned 10 entries:
> > [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_
> ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
> > [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_
> ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_
> 7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_
> EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b)
> [0x7fadba32c01b]
> > [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_
> ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vecto
> rIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
> > [bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245)
> [0x7fadba18f165]
> > [bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c)
> [0x7fadde26cadc]
> > [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc)
> [0x7fadde26c40c]
> > [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
> x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
> > [bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-
> x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
> > [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
> > [bt] (9) /usr/bin/python3() [0x48f2df]
> >
> > -Kellen
> > Amazon Development Center Germany GmbH
> > Berlin - Dresden - Aachen
> > main office: Krausenstr. 38, 10117 Berlin
> > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > Ust-ID: DE289237879
> > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
> >
> >
> >
> >
> > Amazon Development Center Germany GmbH
> > Berlin - Dresden - Aachen
> > main office: Krausenstr. 38, 10117 Berlin
> > Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> > Ust-ID: DE289237879
> > Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>
--
Best Regards,
Gautam Kumar
Re: CI system seems to be using python3 for python2 builds
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Hi Kumar,
Thanks for looking into the issue. How is the progress of this problem?
Shouldn't we call /usr/bin/env python2 or python2.7 in following
source code instead of python since MXNet only supports python2
currently?
I think default version of python in Ubuntu is now python3, so it can
cause the problem.
If you have not yet done the work, I can create a PR for that in this weekend.
./python/mxnet/__init__.py:#!/usr/bin/env python
./python/mxnet/log.py:#!/usr/bin/env python
./tests/nightly/dist_lenet.py:#!/usr/bin/env python
./tests/nightly/dist_sync_kvstore.py:#!/usr/bin/env python
./tests/nightly/multi_lenet.py:#!/usr/bin/env python
./tests/nightly/test_kvstore.py:#!/usr/bin/env python
./tools/coreml/mxnet_coreml_converter.py:#!/usr/bin/env python
./tools/ipynb2md.py:#!/usr/bin/env python
./tools/kill-mxnet.py:#!/usr/bin/env python
./tools/launch.py:#!/usr/bin/env python
./tools/parse_log.py:#!/usr/bin/env python
On Wed, Sep 27, 2017 at 5:39 PM, Sunderland, Kellen <ke...@amazon.de> wrote:
> Many thanks Gautam.
>
> On 9/26/17, 8:37 PM, "Kumar, Gautam" <ga...@amazon.com> wrote:
>
> Hi Kellen,
>
> This issue has been happening since last 3-4 days along with few other test failure.
> I am looking into it.
>
> -Gautam
>
> On 9/26/17, 7:45 AM, "Sunderland, Kellen" <ke...@amazon.de> wrote:
>
> I’ve been noticing in a few failed builds that the stack trace indicates we’re actually running python 3.4 in the python 2 tests. I know the CI folks are working hard getting everything setup, is this a known issue for the CI team?
>
> For example: https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
>
> Steps Python2: MKLML-CPU
>
> StackTrace:
> Stack trace returned 10 entries:
> [bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
> [bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b) [0x7fadba32c01b]
> [bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vectorIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
> [bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245) [0x7fadba18f165]
> [bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fadde26cadc]
> [bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fadde26c40c]
> [bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
> [bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
> [bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
> [bt] (9) /usr/bin/python3() [0x48f2df]
>
> -Kellen
> Amazon Development Center Germany GmbH
> Berlin - Dresden - Aachen
> main office: Krausenstr. 38, 10117 Berlin
> Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> Ust-ID: DE289237879
> Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
>
>
>
>
> Amazon Development Center Germany GmbH
> Berlin - Dresden - Aachen
> main office: Krausenstr. 38, 10117 Berlin
> Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
> Ust-ID: DE289237879
> Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
Re: CI system seems to be using python3 for python2 builds
Posted by "Sunderland, Kellen" <ke...@amazon.de>.
Many thanks Gautam.
On 9/26/17, 8:37 PM, "Kumar, Gautam" <ga...@amazon.com> wrote:
Hi Kellen,
This issue has been happening since last 3-4 days along with few other test failure.
I am looking into it.
-Gautam
On 9/26/17, 7:45 AM, "Sunderland, Kellen" <ke...@amazon.de> wrote:
I’ve been noticing in a few failed builds that the stack trace indicates we’re actually running python 3.4 in the python 2 tests. I know the CI folks are working hard getting everything setup, is this a known issue for the CI team?
For example: https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
Steps Python2: MKLML-CPU
StackTrace:
Stack trace returned 10 entries:
[bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
[bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b) [0x7fadba32c01b]
[bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vectorIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
[bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245) [0x7fadba18f165]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fadde26cadc]
[bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fadde26c40c]
[bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
[bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
[bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
[bt] (9) /usr/bin/python3() [0x48f2df]
-Kellen
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B
Re: CI system seems to be using python3 for python2 builds
Posted by "Kumar, Gautam" <ga...@amazon.com>.
Hi Kellen,
This issue has been happening since last 3-4 days along with few other test failure.
I am looking into it.
-Gautam
On 9/26/17, 7:45 AM, "Sunderland, Kellen" <ke...@amazon.de> wrote:
I’ve been noticing in a few failed builds that the stack trace indicates we’re actually running python 3.4 in the python 2 tests. I know the CI folks are working hard getting everything setup, is this a known issue for the CI team?
For example: https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/PR-8026/3/pipeline/281
Steps Python2: MKLML-CPU
StackTrace:
Stack trace returned 10 entries:
[bt] (0) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fadb8999aac]
[bt] (1) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal12GroupKVPairsISt4pairIPNS_7NDArrayES4_EZNS1_19GroupKVPairsPullRspERKSt6vectorIiSaIiEERKS7_IS6_SaIS6_EEPS9_PS7_ISD_SaISD_EEEUliRKS6_E_EEvSB_RKS7_IT_SaISN_EESG_PS7_ISP_SaISP_EERKT0_+0x56b) [0x7fadba32c01b]
[bt] (2) /workspace/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7kvstore12KVStoreLocal17PullRowSparseImplERKSt6vectorIiSaIiEERKS2_ISt4pairIPNS_7NDArrayES8_ESaISA_EEi+0xa6) [0x7fadba32c856]
[bt] (3) /workspace/python/mxnet/../../lib/libmxnet.so(MXKVStorePullRowSparse+0x245) [0x7fadba18f165]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fadde26cadc]
[bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fadde26c40c]
[bt] (6) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(_ctypes_callproc+0x21d) [0x7fadde47e12d]
[bt] (7) /usr/lib/python3.4/lib-dynload/_ctypes.cpython-34m-x86_64-linux-gnu.so(+0xf6a3) [0x7fadde47e6a3]
[bt] (8) /usr/bin/python3(PyEval_EvalFrameEx+0x41d7) [0x48a487]
[bt] (9) /usr/bin/python3() [0x48f2df]
-Kellen
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B