You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ravindra Pindikura <ra...@dremio.com> on 2018/12/25 18:53:15 UTC
Debugging Gandiva failures in python CI tests
Hi,
My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
https://travis-ci.org/apache/arrow/jobs/471874254#L7167
However, the travis log has no further information on the failure.
To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
I got stuck at this (virtualenv pyarrow):
ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
Is this a known issue ? Any workaround ?
Thanks & regards,
Ravindra.
Re: Debugging Gandiva failures in python CI tests
Posted by Antoine Pitrou <an...@python.org>.
IMHO we can simpler skip the Gandiva tests on Python 2.7.
Regards
Antoine.
Le 07/01/2019 à 21:12, Ravindra Pindikura a écrit :
> Thanks Antoine.
>
> This works on my test machine. But fails in CI. I’m guessing it’s a python 2.7 issue.
>
> https://travis-ci.org/apache/arrow/jobs/476469678 <https://travis-ci.org/apache/arrow/jobs/476469678>
>
>
> ImportError while loading conftest '/home/travis/build/apache/arrow/python/pyarrow/tests/conftest.py'.
> <>../pyarrow-test-2.7/lib/python2.7/site-packages/six.py:709: in exec_
> <> exec("""exec _code_ in _globs_, _locs_""")
> <>pyarrow/tests/conftest.py:66: in <module>
> <> import pyarrow.gandiva # noqa
> <>pyarrow/gandiva.pyx:85: in init pyarrow.gandiva
> <> _dll = ctypes.CDLL(__file__, ctypes.RTLD_GLOBAL)
> <>E NameError: name '__file__' is not defined
>
>
>
>
>
>> On Jan 7, 2019, at 8:05 PM, Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Hi Ravindra,
>>
>> I've found that it's the same issue as previously in Java.
>> Here is a workaround that works here:
>> https://gist.github.com/pitrou/7eed7b980eec3187d619bc146b75cc4b
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 07/01/2019 à 07:16, Ravindra Pindikura a écrit :
>>> I missed mentioning this. The repro requires my changes for decimal (in the PR).
>>>
>>> https://github.com/apache/arrow/pull/2942 <https://github.com/apache/arrow/pull/2942>
>
>
Re: Debugging Gandiva failures in python CI tests
Posted by Ravindra Pindikura <ra...@dremio.com>.
Thanks Antoine.
This works on my test machine. But fails in CI. I’m guessing it’s a python 2.7 issue.
https://travis-ci.org/apache/arrow/jobs/476469678 <https://travis-ci.org/apache/arrow/jobs/476469678>
ImportError while loading conftest '/home/travis/build/apache/arrow/python/pyarrow/tests/conftest.py'.
<>../pyarrow-test-2.7/lib/python2.7/site-packages/six.py:709: in exec_
<> exec("""exec _code_ in _globs_, _locs_""")
<>pyarrow/tests/conftest.py:66: in <module>
<> import pyarrow.gandiva # noqa
<>pyarrow/gandiva.pyx:85: in init pyarrow.gandiva
<> _dll = ctypes.CDLL(__file__, ctypes.RTLD_GLOBAL)
<>E NameError: name '__file__' is not defined
> On Jan 7, 2019, at 8:05 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> Hi Ravindra,
>
> I've found that it's the same issue as previously in Java.
> Here is a workaround that works here:
> https://gist.github.com/pitrou/7eed7b980eec3187d619bc146b75cc4b
>
> Regards
>
> Antoine.
>
>
> Le 07/01/2019 à 07:16, Ravindra Pindikura a écrit :
>> I missed mentioning this. The repro requires my changes for decimal (in the PR).
>>
>> https://github.com/apache/arrow/pull/2942 <https://github.com/apache/arrow/pull/2942>
Re: Debugging Gandiva failures in python CI tests
Posted by Antoine Pitrou <an...@python.org>.
Hi Ravindra,
I've found that it's the same issue as previously in Java.
Here is a workaround that works here:
https://gist.github.com/pitrou/7eed7b980eec3187d619bc146b75cc4b
Regards
Antoine.
Le 07/01/2019 à 07:16, Ravindra Pindikura a écrit :
> I missed mentioning this. The repro requires my changes for decimal (in the PR).
>
> https://github.com/apache/arrow/pull/2942 <https://github.com/apache/arrow/pull/2942>
Re: Debugging Gandiva failures in python CI tests
Posted by Ravindra Pindikura <ra...@dremio.com>.
I missed mentioning this. The repro requires my changes for decimal (in the PR).
https://github.com/apache/arrow/pull/2942 <https://github.com/apache/arrow/pull/2942>
> On Jan 6, 2019, at 4:23 PM, Ravindra Pindikura <ra...@dremio.com> wrote:
>
>
>
>> On Jan 6, 2019, at 3:38 PM, Antoine Pitrou <antoine@python.org <ma...@python.org>> wrote:
>>
>>
>> Can you tell me how to reproduce? I'll take a look on Monday.
>
>
> Thank you. I used the following steps to repro (on ubuntu 14) :
>
> 1. conda section to setup the python environment
>
> https://github.com/apache/arrow/blob/master/docs/source/python/development.rst <https://github.com/apache/arrow/blob/master/docs/source/python/development.rst>
>
> 2. Build python gandiva tests
>
> python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-gandiva —inplace
>
> 3. Run python gandiva tests
>
> pytest pyarrow --only-gandiva
>
> The test silently dies without printing any error msg. So, I modified test_gandiva.py to use pdb and stepped through till I got the error (pdb.set_trace() in test_tree_expr_builder()).
>
> Thanks & regards,
> Ravindra.
>
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 06/01/2019 à 10:24, Ravindra Pindikura a écrit :
>>>
>>>
>>>> On Dec 29, 2018, at 8:03 PM, Ravindra Pindikura <ravindra@dremio.com <ma...@dremio.com>> wrote:
>>>>
>>>>
>>>>
>>>>> On Dec 28, 2018, at 1:38 AM, Wes McKinney <wesmckinn@gmail.com <ma...@gmail.com>> wrote:
>>>>>
>>>>> You might have to wait for Antoine's help after the holidays. In the
>>>>> meantime: do you get the error when building both with gcc 4.8/4.9 and
>>>>> clang-6.0? I did a bit of google searching and it seems that iostream
>>>>> is a common troublemaker for LLVM. I wonder if the code can be
>>>>> refactored to avoid using std::cout
>>>>
>>>> Thanks Wes.
>>>>
>>>> I’ll wait for Antoine. In the meanwhile, I’ll work on the refactoring, just in case.
>>>
>>> Antoine,
>>>
>>> Can you please give your opinion on this ? I’ve hit this with java and python. Fairly sure it’ll be a problem on windows too.
>>>
>>> Should I just break down arrow/util/decimal.cc <http://decimal.cc/> <http://decimal.cc/ <http://decimal.cc/>> into two parts : one that works with llvm ir (no iostream), and the remaining
>>>
>>> Thanks & regards,
>>> ravindra.
>>>
>>>>
>>>>
>>>>>
>>>>> On Thu, Dec 27, 2018 at 3:56 AM Ravindra Pindikura <ravindra@dremio.com <ma...@dremio.com>> wrote:
>>>>>>
>>>>>> Thanks Wes, that worked.
>>>>>>
>>>>>> When I run the gandiva tests (I did a step-through with pdb), I get this error :
>>>>>>
>>>>>>> /home/ravindra/git/arrow/python/pyarrow/tests/test_gandiva.py(50)test_tree_exp_builder()
>>>>>> -> schema, [expr], pa.default_memory_pool())
>>>>>> (Pdb)
>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>
>>>>>> I had hit the same issue with java/jni, and resolved it by doing a NativeLibrary.getInstance() (thanks to @pitrou). My knowledge of python is limited - can someone please help with the same for python ?
>>>>>>
>>>>>> Thanks & regards,
>>>>>> ravindra.
>>>>>>
>>>>>>> On Dec 26, 2018, at 7:55 PM, Wes McKinney <wesmckinn@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>> hi Ravindra,
>>>>>>>
>>>>>>> If you are using Anaconda, you need to use the conda instructions for
>>>>>>> environment setup instead of virtualenv. This should be called out
>>>>>>> more prominently in a warning in the docs
>>>>>>>
>>>>>>> - Wes
>>>>>>>
>>>>>>> On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ravindra@dremio.com <ma...@dremio.com>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
>>>>>>>>
>>>>>>>> https://travis-ci.org/apache/arrow/jobs/471874254#L7167 <https://travis-ci.org/apache/arrow/jobs/471874254#L7167>
>>>>>>>>
>>>>>>>> However, the travis log has no further information on the failure.
>>>>>>>>
>>>>>>>> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
>>>>>>>>
>>>>>>>> I got stuck at this (virtualenv pyarrow):
>>>>>>>>
>>>>>>>> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
>>>>>>>>
>>>>>>>>
>>>>>>>> Is this a known issue ? Any workaround ?
>>>>>>>>
>>>>>>>> Thanks & regards,
>>>>>>>> Ravindra.
>
Re: Debugging Gandiva failures in python CI tests
Posted by Ravindra Pindikura <ra...@dremio.com>.
> On Jan 6, 2019, at 3:38 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> Can you tell me how to reproduce? I'll take a look on Monday.
Thank you. I used the following steps to repro (on ubuntu 14) :
1. conda section to setup the python environment
https://github.com/apache/arrow/blob/master/docs/source/python/development.rst <https://github.com/apache/arrow/blob/master/docs/source/python/development.rst>
2. Build python gandiva tests
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-gandiva —inplace
3. Run python gandiva tests
pytest pyarrow --only-gandiva
The test silently dies without printing any error msg. So, I modified test_gandiva.py to use pdb and stepped through till I got the error (pdb.set_trace() in test_tree_expr_builder()).
Thanks & regards,
Ravindra.
>
> Regards
>
> Antoine.
>
>
> Le 06/01/2019 à 10:24, Ravindra Pindikura a écrit :
>>
>>
>>> On Dec 29, 2018, at 8:03 PM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>
>>>
>>>
>>>> On Dec 28, 2018, at 1:38 AM, Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>> You might have to wait for Antoine's help after the holidays. In the
>>>> meantime: do you get the error when building both with gcc 4.8/4.9 and
>>>> clang-6.0? I did a bit of google searching and it seems that iostream
>>>> is a common troublemaker for LLVM. I wonder if the code can be
>>>> refactored to avoid using std::cout
>>>
>>> Thanks Wes.
>>>
>>> I’ll wait for Antoine. In the meanwhile, I’ll work on the refactoring, just in case.
>>
>> Antoine,
>>
>> Can you please give your opinion on this ? I’ve hit this with java and python. Fairly sure it’ll be a problem on windows too.
>>
>> Should I just break down arrow/util/decimal.cc <http://decimal.cc/> <http://decimal.cc/ <http://decimal.cc/>> into two parts : one that works with llvm ir (no iostream), and the remaining
>>
>> Thanks & regards,
>> ravindra.
>>
>>>
>>>
>>>>
>>>> On Thu, Dec 27, 2018 at 3:56 AM Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>
>>>>> Thanks Wes, that worked.
>>>>>
>>>>> When I run the gandiva tests (I did a step-through with pdb), I get this error :
>>>>>
>>>>>> /home/ravindra/git/arrow/python/pyarrow/tests/test_gandiva.py(50)test_tree_exp_builder()
>>>>> -> schema, [expr], pa.default_memory_pool())
>>>>> (Pdb)
>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>
>>>>> I had hit the same issue with java/jni, and resolved it by doing a NativeLibrary.getInstance() (thanks to @pitrou). My knowledge of python is limited - can someone please help with the same for python ?
>>>>>
>>>>> Thanks & regards,
>>>>> ravindra.
>>>>>
>>>>>> On Dec 26, 2018, at 7:55 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>>
>>>>>> hi Ravindra,
>>>>>>
>>>>>> If you are using Anaconda, you need to use the conda instructions for
>>>>>> environment setup instead of virtualenv. This should be called out
>>>>>> more prominently in a warning in the docs
>>>>>>
>>>>>> - Wes
>>>>>>
>>>>>> On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
>>>>>>>
>>>>>>> https://travis-ci.org/apache/arrow/jobs/471874254#L7167
>>>>>>>
>>>>>>> However, the travis log has no further information on the failure.
>>>>>>>
>>>>>>> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
>>>>>>>
>>>>>>> I got stuck at this (virtualenv pyarrow):
>>>>>>>
>>>>>>> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
>>>>>>>
>>>>>>>
>>>>>>> Is this a known issue ? Any workaround ?
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Ravindra.
Re: Debugging Gandiva failures in python CI tests
Posted by Antoine Pitrou <an...@python.org>.
Can you tell me how to reproduce? I'll take a look on Monday.
Regards
Antoine.
Le 06/01/2019 à 10:24, Ravindra Pindikura a écrit :
>
>
>> On Dec 29, 2018, at 8:03 PM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>
>>
>>
>>> On Dec 28, 2018, at 1:38 AM, Wes McKinney <we...@gmail.com> wrote:
>>>
>>> You might have to wait for Antoine's help after the holidays. In the
>>> meantime: do you get the error when building both with gcc 4.8/4.9 and
>>> clang-6.0? I did a bit of google searching and it seems that iostream
>>> is a common troublemaker for LLVM. I wonder if the code can be
>>> refactored to avoid using std::cout
>>
>> Thanks Wes.
>>
>> I’ll wait for Antoine. In the meanwhile, I’ll work on the refactoring, just in case.
>
> Antoine,
>
> Can you please give your opinion on this ? I’ve hit this with java and python. Fairly sure it’ll be a problem on windows too.
>
> Should I just break down arrow/util/decimal.cc <http://decimal.cc/> into two parts : one that works with llvm ir (no iostream), and the remaining
>
> Thanks & regards,
> ravindra.
>
>>
>>
>>>
>>> On Thu, Dec 27, 2018 at 3:56 AM Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>
>>>> Thanks Wes, that worked.
>>>>
>>>> When I run the gandiva tests (I did a step-through with pdb), I get this error :
>>>>
>>>>> /home/ravindra/git/arrow/python/pyarrow/tests/test_gandiva.py(50)test_tree_exp_builder()
>>>> -> schema, [expr], pa.default_memory_pool())
>>>> (Pdb)
>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>
>>>> I had hit the same issue with java/jni, and resolved it by doing a NativeLibrary.getInstance() (thanks to @pitrou). My knowledge of python is limited - can someone please help with the same for python ?
>>>>
>>>> Thanks & regards,
>>>> ravindra.
>>>>
>>>>> On Dec 26, 2018, at 7:55 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>>
>>>>> hi Ravindra,
>>>>>
>>>>> If you are using Anaconda, you need to use the conda instructions for
>>>>> environment setup instead of virtualenv. This should be called out
>>>>> more prominently in a warning in the docs
>>>>>
>>>>> - Wes
>>>>>
>>>>> On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
>>>>>>
>>>>>> https://travis-ci.org/apache/arrow/jobs/471874254#L7167
>>>>>>
>>>>>> However, the travis log has no further information on the failure.
>>>>>>
>>>>>> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
>>>>>>
>>>>>> I got stuck at this (virtualenv pyarrow):
>>>>>>
>>>>>> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
>>>>>>
>>>>>>
>>>>>> Is this a known issue ? Any workaround ?
>>>>>>
>>>>>> Thanks & regards,
>>>>>> Ravindra.
>>>>>>
>>>>>>
>>>>
>>
>
>
Re: Debugging Gandiva failures in python CI tests
Posted by Ravindra Pindikura <ra...@dremio.com>.
> On Dec 29, 2018, at 8:03 PM, Ravindra Pindikura <ra...@dremio.com> wrote:
>
>
>
>> On Dec 28, 2018, at 1:38 AM, Wes McKinney <we...@gmail.com> wrote:
>>
>> You might have to wait for Antoine's help after the holidays. In the
>> meantime: do you get the error when building both with gcc 4.8/4.9 and
>> clang-6.0? I did a bit of google searching and it seems that iostream
>> is a common troublemaker for LLVM. I wonder if the code can be
>> refactored to avoid using std::cout
>
> Thanks Wes.
>
> I’ll wait for Antoine. In the meanwhile, I’ll work on the refactoring, just in case.
Antoine,
Can you please give your opinion on this ? I’ve hit this with java and python. Fairly sure it’ll be a problem on windows too.
Should I just break down arrow/util/decimal.cc <http://decimal.cc/> into two parts : one that works with llvm ir (no iostream), and the remaining
Thanks & regards,
ravindra.
>
>
>>
>> On Thu, Dec 27, 2018 at 3:56 AM Ravindra Pindikura <ra...@dremio.com> wrote:
>>>
>>> Thanks Wes, that worked.
>>>
>>> When I run the gandiva tests (I did a step-through with pdb), I get this error :
>>>
>>>> /home/ravindra/git/arrow/python/pyarrow/tests/test_gandiva.py(50)test_tree_exp_builder()
>>> -> schema, [expr], pa.default_memory_pool())
>>> (Pdb)
>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>
>>> I had hit the same issue with java/jni, and resolved it by doing a NativeLibrary.getInstance() (thanks to @pitrou). My knowledge of python is limited - can someone please help with the same for python ?
>>>
>>> Thanks & regards,
>>> ravindra.
>>>
>>>> On Dec 26, 2018, at 7:55 PM, Wes McKinney <we...@gmail.com> wrote:
>>>>
>>>> hi Ravindra,
>>>>
>>>> If you are using Anaconda, you need to use the conda instructions for
>>>> environment setup instead of virtualenv. This should be called out
>>>> more prominently in a warning in the docs
>>>>
>>>> - Wes
>>>>
>>>> On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
>>>>>
>>>>> https://travis-ci.org/apache/arrow/jobs/471874254#L7167
>>>>>
>>>>> However, the travis log has no further information on the failure.
>>>>>
>>>>> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
>>>>>
>>>>> I got stuck at this (virtualenv pyarrow):
>>>>>
>>>>> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
>>>>>
>>>>>
>>>>> Is this a known issue ? Any workaround ?
>>>>>
>>>>> Thanks & regards,
>>>>> Ravindra.
>>>>>
>>>>>
>>>
>
Re: Debugging Gandiva failures in python CI tests
Posted by Ravindra Pindikura <ra...@dremio.com>.
> On Dec 28, 2018, at 1:38 AM, Wes McKinney <we...@gmail.com> wrote:
>
> You might have to wait for Antoine's help after the holidays. In the
> meantime: do you get the error when building both with gcc 4.8/4.9 and
> clang-6.0? I did a bit of google searching and it seems that iostream
> is a common troublemaker for LLVM. I wonder if the code can be
> refactored to avoid using std::cout
Thanks Wes.
I’ll wait for Antoine. In the meanwhile, I’ll work on the refactoring, just in case.
>
> On Thu, Dec 27, 2018 at 3:56 AM Ravindra Pindikura <ra...@dremio.com> wrote:
>>
>> Thanks Wes, that worked.
>>
>> When I run the gandiva tests (I did a step-through with pdb), I get this error :
>>
>>> /home/ravindra/git/arrow/python/pyarrow/tests/test_gandiva.py(50)test_tree_exp_builder()
>> -> schema, [expr], pa.default_memory_pool())
>> (Pdb)
>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>
>> I had hit the same issue with java/jni, and resolved it by doing a NativeLibrary.getInstance() (thanks to @pitrou). My knowledge of python is limited - can someone please help with the same for python ?
>>
>> Thanks & regards,
>> ravindra.
>>
>>> On Dec 26, 2018, at 7:55 PM, Wes McKinney <we...@gmail.com> wrote:
>>>
>>> hi Ravindra,
>>>
>>> If you are using Anaconda, you need to use the conda instructions for
>>> environment setup instead of virtualenv. This should be called out
>>> more prominently in a warning in the docs
>>>
>>> - Wes
>>>
>>> On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
>>>>
>>>> https://travis-ci.org/apache/arrow/jobs/471874254#L7167
>>>>
>>>> However, the travis log has no further information on the failure.
>>>>
>>>> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
>>>>
>>>> I got stuck at this (virtualenv pyarrow):
>>>>
>>>> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
>>>>
>>>>
>>>> Is this a known issue ? Any workaround ?
>>>>
>>>> Thanks & regards,
>>>> Ravindra.
>>>>
>>>>
>>
Re: Debugging Gandiva failures in python CI tests
Posted by Wes McKinney <we...@gmail.com>.
You might have to wait for Antoine's help after the holidays. In the
meantime: do you get the error when building both with gcc 4.8/4.9 and
clang-6.0? I did a bit of google searching and it seems that iostream
is a common troublemaker for LLVM. I wonder if the code can be
refactored to avoid using std::cout
On Thu, Dec 27, 2018 at 3:56 AM Ravindra Pindikura <ra...@dremio.com> wrote:
>
> Thanks Wes, that worked.
>
> When I run the gandiva tests (I did a step-through with pdb), I get this error :
>
> > /home/ravindra/git/arrow/python/pyarrow/tests/test_gandiva.py(50)test_tree_exp_builder()
> -> schema, [expr], pa.default_memory_pool())
> (Pdb)
> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>
> I had hit the same issue with java/jni, and resolved it by doing a NativeLibrary.getInstance() (thanks to @pitrou). My knowledge of python is limited - can someone please help with the same for python ?
>
> Thanks & regards,
> ravindra.
>
> > On Dec 26, 2018, at 7:55 PM, Wes McKinney <we...@gmail.com> wrote:
> >
> > hi Ravindra,
> >
> > If you are using Anaconda, you need to use the conda instructions for
> > environment setup instead of virtualenv. This should be called out
> > more prominently in a warning in the docs
> >
> > - Wes
> >
> > On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ra...@dremio.com> wrote:
> >>
> >> Hi,
> >>
> >> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
> >>
> >> https://travis-ci.org/apache/arrow/jobs/471874254#L7167
> >>
> >> However, the travis log has no further information on the failure.
> >>
> >> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
> >>
> >> I got stuck at this (virtualenv pyarrow):
> >>
> >> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
> >>
> >>
> >> Is this a known issue ? Any workaround ?
> >>
> >> Thanks & regards,
> >> Ravindra.
> >>
> >>
>
Re: Debugging Gandiva failures in python CI tests
Posted by Ravindra Pindikura <ra...@dremio.com>.
Thanks Wes, that worked.
When I run the gandiva tests (I did a step-through with pdb), I get this error :
> /home/ravindra/git/arrow/python/pyarrow/tests/test_gandiva.py(50)test_tree_exp_builder()
-> schema, [expr], pa.default_memory_pool())
(Pdb)
LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
I had hit the same issue with java/jni, and resolved it by doing a NativeLibrary.getInstance() (thanks to @pitrou). My knowledge of python is limited - can someone please help with the same for python ?
Thanks & regards,
ravindra.
> On Dec 26, 2018, at 7:55 PM, Wes McKinney <we...@gmail.com> wrote:
>
> hi Ravindra,
>
> If you are using Anaconda, you need to use the conda instructions for
> environment setup instead of virtualenv. This should be called out
> more prominently in a warning in the docs
>
> - Wes
>
> On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ra...@dremio.com> wrote:
>>
>> Hi,
>>
>> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
>>
>> https://travis-ci.org/apache/arrow/jobs/471874254#L7167
>>
>> However, the travis log has no further information on the failure.
>>
>> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
>>
>> I got stuck at this (virtualenv pyarrow):
>>
>> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
>>
>>
>> Is this a known issue ? Any workaround ?
>>
>> Thanks & regards,
>> Ravindra.
>>
>>
Re: Debugging Gandiva failures in python CI tests
Posted by Wes McKinney <we...@gmail.com>.
hi Ravindra,
If you are using Anaconda, you need to use the conda instructions for
environment setup instead of virtualenv. This should be called out
more prominently in a warning in the docs
- Wes
On Tue, Dec 25, 2018 at 12:53 PM Ravindra Pindikura <ra...@dremio.com> wrote:
>
> Hi,
>
> My change (decimal support in gandiva) causes a CI failure in python test_gandiva.py
>
> https://travis-ci.org/apache/arrow/jobs/471874254#L7167
>
> However, the travis log has no further information on the failure.
>
> To debug this, I tried setting up a python dev environment on ubuntu 14 using the steps in https://github.com/apache/arrow/blob/master/docs/source/python/development.rst
>
> I got stuck at this (virtualenv pyarrow):
>
> ValueError: failed to parse CPython sys.version: '3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:19:38) \n[GCC 7.3.0]'
>
>
> Is this a known issue ? Any workaround ?
>
> Thanks & regards,
> Ravindra.
>
>