You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2018/12/16 13:03:00 UTC
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
need to load it explicitly in the Java process.
Regards
Antoine.
Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>
>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>
>> Hi,
>>
>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>
>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>
>>
>> I’ve the same test (add two decimals) in three forms :
>>
>> 1. Cpp test executable with shared gandiva lib
>>
>> Works on both ubuntu and Mac
>>
>> 2. Cpp test executable with static gandiva lib
>>
>> Works on both ubuntu and Mac
>>
>> 3. Java test (internally uses jni over static gandiva lib)
>>
>> Works on Mac, but fails on ubuntu !!
>>
>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>
>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>
>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>
>> Thanks & regards,
>> Ravindra.
>
>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Wes McKinney <we...@gmail.com>.
We might have to split up the implementation of Decimal128 into
multiple compilation units based on what parts of it are LLVM
IR-friendly
On Sun, Dec 16, 2018 at 7:04 AM Antoine Pitrou <an...@python.org> wrote:
>
>
> In other words, see if:
> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>
> outputs anything.
>
> Regards
>
> Antoine.
>
>
> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
> >
> > Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
> > need to load it explicitly in the Java process.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
> >> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
> >>
> >>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
> >>>
> >>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
> >>>
> >>>
> >>> I’ve the same test (add two decimals) in three forms :
> >>>
> >>> 1. Cpp test executable with shared gandiva lib
> >>>
> >>> Works on both ubuntu and Mac
> >>>
> >>> 2. Cpp test executable with static gandiva lib
> >>>
> >>> Works on both ubuntu and Mac
> >>>
> >>> 3. Java test (internally uses jni over static gandiva lib)
> >>>
> >>> Works on Mac, but fails on ubuntu !!
> >>>
> >>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
> >>>
> >>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
> >>>
> >>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
> >>>
> >>> Thanks & regards,
> >>> Ravindra.
> >>
> >>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Ravindra Pindikura <ra...@dremio.com>.
> On Dec 17, 2018, at 9:38 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> Apparently you're using System.load() to load the compiled helpers
> functions. It seems it may load the library using RTLD_LOCAL, which
> means it won't be seen from other dynamic libraries loaded in the same
> process.
>
> According to this SO entry, the solution would be to use
> NativeLibrary.getInstance() instead:
Thanks a lot. That worked like a charm !
> https://stackoverflow.com/questions/5425034/java-load-shared-librariees-with-dependencies
>
> Regards
>
> Antoine.
>
>
> Le 17/12/2018 à 16:48, Ravindra Pindikura a écrit :
>>
>>
>>> On Dec 17, 2018, at 6:39 PM, Antoine Pitrou <an...@python.org> wrote:
>>>
>>>
>>> Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
>>> loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
>>> explicitly?
>>
>> We are already doing this inside gandiva.
>>
>> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61>
>>
>>>
>>>
>>> Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
>>>> Antoine,
>>>>
>>>> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>>>>
>>>> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>>
>>>> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>>
>>>> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>>>>
>>>> Of the four sub-cases of the test :
>>>>
>>>> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
>>>> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
>>>> 3. a java executable, the decimal functions invoked from CPP Fns
>>>> 4. a java executable, the decimal functions invoked from JIT code
>>>>
>>>> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>>>>
>>>> Thanks & regards,
>>>> Ravindra.
>>>>
>>>>
>>>>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>>>>
>>>>>
>>>>> In other words, see if:
>>>>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>>>>
>>>>> outputs anything.
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>>>>
>>>>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>>>>
>>>>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>>>>> need to load it explicitly in the Java process.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Antoine.
>>>>>>
>>>>>>
>>>>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>>>>
>>>>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>>>>
>>>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>>>
>>>>>>>>
>>>>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>>>>
>>>>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>>>>
>>>>>>>> Works on both ubuntu and Mac
>>>>>>>>
>>>>>>>> 2. Cpp test executable with static gandiva lib
>>>>>>>>
>>>>>>>> Works on both ubuntu and Mac
>>>>>>>>
>>>>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>>>>
>>>>>>>> Works on Mac, but fails on ubuntu !!
>>>>>>>>
>>>>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>>>>
>>>>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>>>>
>>>>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>>>>
>>>>>>>> Thanks & regards,
>>>>>>>> Ravindra.
>>>>>>>
>>>>>>>
>>>>
>>
>>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Antoine Pitrou <an...@python.org>.
Apparently you're using System.load() to load the compiled helpers
functions. It seems it may load the library using RTLD_LOCAL, which
means it won't be seen from other dynamic libraries loaded in the same
process.
According to this SO entry, the solution would be to use
NativeLibrary.getInstance() instead:
https://stackoverflow.com/questions/5425034/java-load-shared-librariees-with-dependencies
Regards
Antoine.
Le 17/12/2018 à 16:48, Ravindra Pindikura a écrit :
>
>
>> On Dec 17, 2018, at 6:39 PM, Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
>> loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
>> explicitly?
>
> We are already doing this inside gandiva.
>
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61>
>
>>
>>
>> Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
>>> Antoine,
>>>
>>> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>>>
>>> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>
>>> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>
>>> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>>>
>>> Of the four sub-cases of the test :
>>>
>>> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
>>> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
>>> 3. a java executable, the decimal functions invoked from CPP Fns
>>> 4. a java executable, the decimal functions invoked from JIT code
>>>
>>> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>>>
>>> Thanks & regards,
>>> Ravindra.
>>>
>>>
>>>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>>>
>>>>
>>>> In other words, see if:
>>>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>>>
>>>> outputs anything.
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>>>
>>>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>>>> need to load it explicitly in the Java process.
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>>>>
>>>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>>>
>>>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>>>
>>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>>
>>>>>>>
>>>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>>>
>>>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>>>
>>>>>>> Works on both ubuntu and Mac
>>>>>>>
>>>>>>> 2. Cpp test executable with static gandiva lib
>>>>>>>
>>>>>>> Works on both ubuntu and Mac
>>>>>>>
>>>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>>>
>>>>>>> Works on Mac, but fails on ubuntu !!
>>>>>>>
>>>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>>>
>>>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>>>
>>>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Ravindra.
>>>>>>
>>>>>>
>>>
>
>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Ravindra Pindikura <ra...@dremio.com>.
> On Dec 17, 2018, at 6:39 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
> loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
> explicitly?
We are already doing this inside gandiva.
https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61>
>
>
> Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
>> Antoine,
>>
>> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>>
>> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>
>> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>
>> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>>
>> Of the four sub-cases of the test :
>>
>> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
>> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
>> 3. a java executable, the decimal functions invoked from CPP Fns
>> 4. a java executable, the decimal functions invoked from JIT code
>>
>> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>>
>> Thanks & regards,
>> Ravindra.
>>
>>
>>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>>
>>>
>>> In other words, see if:
>>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>>
>>> outputs anything.
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>>
>>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>>> need to load it explicitly in the Java process.
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>>
>>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>>
>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>
>>>>>>
>>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>>
>>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>>
>>>>>> Works on both ubuntu and Mac
>>>>>>
>>>>>> 2. Cpp test executable with static gandiva lib
>>>>>>
>>>>>> Works on both ubuntu and Mac
>>>>>>
>>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>>
>>>>>> Works on Mac, but fails on ubuntu !!
>>>>>>
>>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>>
>>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>>
>>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>>
>>>>>> Thanks & regards,
>>>>>> Ravindra.
>>>>>
>>>>>
>>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Antoine Pitrou <an...@python.org>.
Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
loaded? Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
explicitly?
Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
> Antoine,
>
> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>
> $ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>
> $ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>
> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>
> Of the four sub-cases of the test :
>
> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
> 3. a java executable, the decimal functions invoked from CPP Fns
> 4. a java executable, the decimal functions invoked from JIT code
>
> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>
> Thanks & regards,
> Ravindra.
>
>
>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>>
>>
>> In other words, see if:
>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>
>> outputs anything.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>
>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>> need to load it explicitly in the Java process.
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>
>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>
>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>
>>>>>
>>>>> I’ve the same test (add two decimals) in three forms :
>>>>>
>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>
>>>>> Works on both ubuntu and Mac
>>>>>
>>>>> 2. Cpp test executable with static gandiva lib
>>>>>
>>>>> Works on both ubuntu and Mac
>>>>>
>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>
>>>>> Works on Mac, but fails on ubuntu !!
>>>>>
>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>
>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>
>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>
>>>>> Thanks & regards,
>>>>> Ravindra.
>>>>
>>>>
>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Ravindra Pindikura <ra...@dremio.com>.
Antoine,
I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
$ nm ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
$ nm --defined-only ./debug/libgandiva_jni.so|grep _ZNSt8ios_base4InitC1Ev
0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
Of the four sub-cases of the test :
1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
2. a cpp executable (shared/static), the decimal functions invoked from JIT code
3. a java executable, the decimal functions invoked from CPP Fns
4. a java executable, the decimal functions invoked from JIT code
1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
Thanks & regards,
Ravindra.
> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <an...@python.org> wrote:
>
>
> In other words, see if:
> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>
> outputs anything.
>
> Regards
>
> Antoine.
>
>
> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>
>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>> need to load it explicitly in the Java process.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>
>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>
>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>
>>>>
>>>> I’ve the same test (add two decimals) in three forms :
>>>>
>>>> 1. Cpp test executable with shared gandiva lib
>>>>
>>>> Works on both ubuntu and Mac
>>>>
>>>> 2. Cpp test executable with static gandiva lib
>>>>
>>>> Works on both ubuntu and Mac
>>>>
>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>
>>>> Works on Mac, but fails on ubuntu !!
>>>>
>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>
>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>
>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>
>>>> Thanks & regards,
>>>> Ravindra.
>>>
>>>
Re: Weird symbol resolution error with llvm JIT in gandiva jni
Posted by Antoine Pitrou <an...@python.org>.
In other words, see if:
nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
outputs anything.
Regards
Antoine.
Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>
> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
> need to load it explicitly in the Java process.
>
> Regards
>
> Antoine.
>
>
> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>
>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ra...@dremio.com> wrote:
>>>
>>> Hi,
>>>
>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>
>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>
>>>
>>> I’ve the same test (add two decimals) in three forms :
>>>
>>> 1. Cpp test executable with shared gandiva lib
>>>
>>> Works on both ubuntu and Mac
>>>
>>> 2. Cpp test executable with static gandiva lib
>>>
>>> Works on both ubuntu and Mac
>>>
>>> 3. Java test (internally uses jni over static gandiva lib)
>>>
>>> Works on Mac, but fails on ubuntu !!
>>>
>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>
>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>
>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>
>>> Thanks & regards,
>>> Ravindra.
>>
>>