You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ravindra Pindikura <ra...@dremio.com> on 2018/12/16 03:44:19 UTC
Weird symbol resolution error with llvm JIT in gandiva jni
Hi,
I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
I’ve the same test (add two decimals) in three forms :
1. Cpp test executable with shared gandiva lib
Works on both ubuntu and Mac
2. Cpp test executable with static gandiva lib
Works on both ubuntu and Mac
3. Java test (internally uses jni over static gandiva lib)
Works on Mac, but fails on ubuntu !!
Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
Thanks & regards,
Ravindra.
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Wes McKinney <we...@gmail.com>.
We might have to split up the implementation of Decimal128 into
multiple compilation units based on what parts of it are LLVM
IR-friendly
On Sun, Dec 16, 2018 at 7:04 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> In other words, see if:
> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>
> outputs anything.
>
> Regards
>
> Antoine.
>
>
> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
> >
> > Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
> > need to load it explicitly in the Java process.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
> >> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
> >>
> >>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
> >>>
> >>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
> >>>
> >>>
> >>> I’ve the same test (add two decimals) in three forms :
> >>>
> >>> 1. Cpp test executable with shared gandiva lib
> >>>
> >>> Works on both ubuntu and Mac
> >>>
> >>> 2. Cpp test executable with static gandiva lib
> >>>
> >>> Works on both ubuntu and Mac
> >>>
> >>> 3. Java test (internally uses jni over static gandiva lib)
> >>>
> >>> Works on Mac, but fails on ubuntu !!
> >>>
> >>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
> >>>
> >>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
> >>>
> >>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
> >>>
> >>> Thanks & regards,
> >>> Ravindra.
> >>
> >>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Ravindra Pindikura <ra...@dremio.com>.
> On Dec 17, 2018, at 9:38 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> Apparently you're using System.load() to load the compiled helpers
> functions. It seems it may load the library using RTLD_LOCAL, which
> means it won't be seen from other dynamic libraries loaded in the same
> process.
>
> According to this SO entry, the solution would be to use
> NativeLibrary.getInstance() instead:
Thanks a lot. That worked like a charm !
> https://stackoverflow.com/questions/5425034/java-load-shared-librariees-with-dependencies
>
> Regards
>
> Antoine.
>
>
> Le 17/12/2018 à 16:48, Ravindra Pindikura a écrit :
>>
>>
>>> On Dec 17, 2018, at 6:39 PM, Antoine Pitrou <an...@python.org> wrote:
>>>
>>>
>>> Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
>>> loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
>>> explicitly?
>>
>> We are already doing this inside gandiva.
>>
>> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61>
>>
>>>
>>>
>>> Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
>>>> Antoine,
>>>>
>>>> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>>>>
>>>> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>>
>>>> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>>
>>>> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>>>>
>>>> Of the four sub-cases of the test :
>>>>
>>>> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
>>>> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
>>>> 3. a java executable, the decimal functions invoked from CPP Fns
>>>> 4. a java executable, the decimal functions invoked from JIT code
>>>>
>>>> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>>>>
>>>> Thanks & regards,
>>>> Ravindra.
>>>>
>>>>
>>>>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>>>>
>>>>>
>>>>> In other words, see if:
>>>>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>>>>
>>>>> outputs anything.
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>>>>
>>>>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>>>>
>>>>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>>>>> need to load it explicitly in the Java process.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Antoine.
>>>>>>
>>>>>>
>>>>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>>>>
>>>>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>>>>
>>>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>>>
>>>>>>>>
>>>>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>>>>
>>>>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>>>>
>>>>>>>> Works on both ubuntu and Mac
>>>>>>>>
>>>>>>>> 2. Cpp test executable with static gandiva lib
>>>>>>>>
>>>>>>>> Works on both ubuntu and Mac
>>>>>>>>
>>>>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>>>>
>>>>>>>> Works on Mac, but fails on ubuntu !!
>>>>>>>>
>>>>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>>>>
>>>>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>>>>
>>>>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>>>>
>>>>>>>> Thanks & regards,
>>>>>>>> Ravindra.
>>>>>>>
>>>>>>>
>>>>
>>
>>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Antoine Pitrou <an...@python.org>.
Apparently you're using System.load() to load the compiled helpers
functions. It seems it may load the library using RTLD_LOCAL, which
means it won't be seen from other dynamic libraries loaded in the same
process.
According to this SO entry, the solution would be to use
NativeLibrary.getInstance() instead:
https://stackoverflow.com/questions/5425034/java-load-shared-librariees-with-dependencies
Regards
Antoine.
Le 17/12/2018 à 16:48, Ravindra Pindikura a écrit :
>
>
>> On Dec 17, 2018, at 6:39 PM, Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
>> loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
>> explicitly?
>
> We are already doing this inside gandiva.
>
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61>
>
>>
>>
>> Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
>>> Antoine,
>>>
>>> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>>>
>>> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>
>>> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>
>>> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>>>
>>> Of the four sub-cases of the test :
>>>
>>> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
>>> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
>>> 3. a java executable, the decimal functions invoked from CPP Fns
>>> 4. a java executable, the decimal functions invoked from JIT code
>>>
>>> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>>>
>>> Thanks & regards,
>>> Ravindra.
>>>
>>>
>>>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>>>
>>>>
>>>> In other words, see if:
>>>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>>>
>>>> outputs anything.
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>>>
>>>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>>>> need to load it explicitly in the Java process.
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>>>>
>>>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>>>
>>>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>>>
>>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>>
>>>>>>>
>>>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>>>
>>>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>>>
>>>>>>> Works on both ubuntu and Mac
>>>>>>>
>>>>>>> 2. Cpp test executable with static gandiva lib
>>>>>>>
>>>>>>> Works on both ubuntu and Mac
>>>>>>>
>>>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>>>
>>>>>>> Works on Mac, but fails on ubuntu !!
>>>>>>>
>>>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>>>
>>>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>>>
>>>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Ravindra.
>>>>>>
>>>>>>
>>>
>
>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Ravindra Pindikura <ra...@dremio.com>.
> On Dec 17, 2018, at 6:39 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
> loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
> explicitly?
We are already doing this inside gandiva.
https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61>
>
>
> Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
>> Antoine,
>>
>> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>>
>> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>
>> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>
>> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>>
>> Of the four sub-cases of the test :
>>
>> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
>> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
>> 3. a java executable, the decimal functions invoked from CPP Fns
>> 4. a java executable, the decimal functions invoked from JIT code
>>
>> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>>
>> Thanks & regards,
>> Ravindra.
>>
>>
>>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>>
>>>
>>> In other words, see if:
>>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>>
>>> outputs anything.
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>>
>>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>>> need to load it explicitly in the Java process.
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>>
>>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>>
>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>
>>>>>>
>>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>>
>>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>>
>>>>>> Works on both ubuntu and Mac
>>>>>>
>>>>>> 2. Cpp test executable with static gandiva lib
>>>>>>
>>>>>> Works on both ubuntu and Mac
>>>>>>
>>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>>
>>>>>> Works on Mac, but fails on ubuntu !!
>>>>>>
>>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>>
>>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>>
>>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>>
>>>>>> Thanks & regards,
>>>>>> Ravindra.
>>>>>
>>>>>
>>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Antoine Pitrou <an...@python.org>.
Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
explicitly?
Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
> Antoine,
>
> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>
> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>
> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>
> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>
> Of the four sub-cases of the test :
>
> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
> 3. a java executable, the decimal functions invoked from CPP Fns
> 4. a java executable, the decimal functions invoked from JIT code
>
> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>
> Thanks & regards,
> Ravindra.
>
>
>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> In other words, see if:
>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>
>> outputs anything.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>
>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>> need to load it explicitly in the Java process.
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>
>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>
>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>
>>>>>
>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>
>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>
>>>>> Works on both ubuntu and Mac
>>>>>
>>>>> 2. Cpp test executable with static gandiva lib
>>>>>
>>>>> Works on both ubuntu and Mac
>>>>>
>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>
>>>>> Works on Mac, but fails on ubuntu !!
>>>>>
>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>
>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>
>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>
>>>>> Thanks & regards,
>>>>> Ravindra.
>>>>
>>>>
>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Ravindra Pindikura <ra...@dremio.com>.
Antoine,
I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
$ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
$ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
Of the four sub-cases of the test :
1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
2. a cpp executable (shared/static), the decimal functions invoked from JIT code
3. a java executable, the decimal functions invoked from CPP Fns
4. a java executable, the decimal functions invoked from JIT code
1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
Thanks & regards,
Ravindra.
> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> In other words, see if:
> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>
> outputs anything.
>
> Regards
>
> Antoine.
>
>
> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>
>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>> need to load it explicitly in the Java process.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>
>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>
>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>
>>>>
>>>> I’ve the same test (add two decimals) in three forms :
>>>>
>>>> 1. Cpp test executable with shared gandiva lib
>>>>
>>>> Works on both ubuntu and Mac
>>>>
>>>> 2. Cpp test executable with static gandiva lib
>>>>
>>>> Works on both ubuntu and Mac
>>>>
>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>
>>>> Works on Mac, but fails on ubuntu !!
>>>>
>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>
>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>
>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>
>>>> Thanks & regards,
>>>> Ravindra.
>>>
>>>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Antoine Pitrou <an...@python.org>.
In other words, see if:
nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
outputs anything.
Regards
Antoine.
Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>
> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
> need to load it explicitly in the Java process.
>
> Regards
>
> Antoine.
>
>
> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>
>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>
>>> Hi,
>>>
>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>
>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>
>>>
>>> I’ve the same test (add two decimals) in three forms :
>>>
>>> 1. Cpp test executable with shared gandiva lib
>>>
>>> Works on both ubuntu and Mac
>>>
>>> 2. Cpp test executable with static gandiva lib
>>>
>>> Works on both ubuntu and Mac
>>>
>>> 3. Java test (internally uses jni over static gandiva lib)
>>>
>>> Works on Mac, but fails on ubuntu !!
>>>
>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>
>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>
>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>
>>> Thanks & regards,
>>> Ravindra.
>>
>>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Antoine Pitrou <an...@python.org>.
Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
need to load it explicitly in the Java process.
Regards
Antoine.
Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>
>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>
>> Hi,
>>
>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>
>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>
>>
>> I’ve the same test (add two decimals) in three forms :
>>
>> 1. Cpp test executable with shared gandiva lib
>>
>> Works on both ubuntu and Mac
>>
>> 2. Cpp test executable with static gandiva lib
>>
>> Works on both ubuntu and Mac
>>
>> 3. Java test (internally uses jni over static gandiva lib)
>>
>> Works on Mac, but fails on ubuntu !!
>>
>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>
>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>
>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>
>> Thanks & regards,
>> Ravindra.
>
>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Ravindra Pindikura <ra...@dremio.com>.
I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>
> Hi,
>
> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>
> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>
>
> I’ve the same test (add two decimals) in three forms :
>
> 1. Cpp test executable with shared gandiva lib
>
> Works on both ubuntu and Mac
>
> 2. Cpp test executable with static gandiva lib
>
> Works on both ubuntu and Mac
>
> 3. Java test (internally uses jni over static gandiva lib)
>
> Works on Mac, but fails on ubuntu !!
>
> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>
> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>
> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>
> Thanks & regards,
> Ravindra.