You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Russell Bateman <ru...@windofkeltia.com> on 2020/10/13 21:35:59 UTC
Hive NAR not loading because of snappy?
Should I be seeing this in the log of a vanilla NiFi installation on CentOS?
ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to
java.util.ServiceConfigurationError:
org.apache.nifi.processor.Processor: Provider
org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized
.
.
.
Caused by: org.xerial.snappy.SnappyError:
[FAILED_TO_LOAD_NATIVE_LIBRARY] null
"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the
wrong /nifi-1.11.4-bin.zip/?
Re: Hive NAR not loading because of snappy?
Posted by Russell Bateman <ru...@windofkeltia.com>.
Our current installations are running on NiFi 1.1.2, I think. Recently,
the company wanted to move up in the world. So, we don't have that
continuous experience. We started on 0.7.1.
We're running Java 8 on CentOS 8, I think.
Personally, I run Linux Mint (Ubuntu, corrected) for development. I have
never before had any reason to think that NiFi on CentOS was any
different than on Fedora, Ubuntu or Mint (among the Linux boxes I have
run on over the last 4-5 years).
On 10/13/20 6:35 PM, Matt Burgess wrote:
> Understood. But outside of a permissions issue or usage of
> CompressContent or something else that would try to load the same
> version of Snappy, I’m not sure what’s going on there. AFAIK other
> folks install this on centos without issue. The related snappy issues
> were supposedly resolved well before 1.11.4. Did it used to work on
> previous versions? Have you tried the latest release to see if it
> works? Has the version of Java changed?
>
> Sent from my iPhone
>
>> On Oct 13, 2020, at 8:13 PM, Russell Bateman <ru...@windofkeltia.com>
>> wrote:
>>
>> We didn't want to make deletion or modification of a shipped
>> component to be a required part of its installation since we don't
>> produce the NiFi download. We'd rather install it as it comes.
>>
>> On 10/13/20 4:45 PM, Matt Burgess wrote:
>>> Ouch! It does happen on the loading of the NAR to ensure the native
>>> library gets loaded. If you are not using Hive I’d think you could
>>> safely delete the nifi-hive-nar and it shouldn’t happen. Hard to
>>> tell why the native library couldn’t be installed though.
>>>
>>>
>>>> On Oct 13, 2020, at 6:26 PM, Russell Bateman
>>>> <ru...@windofkeltia.com> wrote:
>>>>
>>>> No, we don't even use (nor have we ever used) Hive in our flows.
>>>> It's just there and we didn't want to modify the NiFi download.
>>>> Should this not even happen if we're not using it?
>>>>
>>>> On 10/13/20 4:24 PM, Matt Burgess wrote:
>>>>> Ugh now I remember, that version of Hive uses a version of Snappy
>>>>> that doesn’t create a unique path under /tmp, do you have multiple
>>>>> PutHiveStreaming processors in the flow? I don’t think that works
>>>>> because we can’t load a single native library into multiple
>>>>> classloaders.
>>>>>
>>>>>
>>>>>> On Oct 13, 2020, at 6:15 PM, Russell Bateman
>>>>>> <ru...@windofkeltia.com> wrote:
>>>>>>
>>>>>> I see
>>>>>>
>>>>>> -rwxr-xr-x. 1 nifi nifi 48432 Oct 13 13:48
>>>>>> snappy-1.0.5-libsnappyjava.so
>>>>>>
>>>>>> in //tmp/. Therefore, not a permissions issue? Launching this way
>>>>>> works:
>>>>>>
>>>>>> $ ( export
>>>>>> LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so"
>>>>>> && /opt/nifi/bin/nifi.sh start )
>>>>>>
>>>>>> but it's not something we want to do (in case that shared object
>>>>>> disappears from //tmp/).
>>>>>>
>>>>>>
>>>>>> On 10/13/20 3:42 PM, Matt Burgess wrote:
>>>>>>> IIRC this is likely a permissions issue, Xerial Snappy tries to
>>>>>>> unzip the native library to the location pointed to by
>>>>>>> “java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi
>>>>>>> user have write access to that directory? If not you can change
>>>>>>> the Java temp dir or set it specifically for Snappy (I don’t
>>>>>>> have the property on hand but a quick Google should find it)
>>>>>>>
>>>>>>> Regards,
>>>>>>> Matt
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>>> On Oct 13, 2020, at 5:36 PM, Russell Bateman
>>>>>>>> <ru...@windofkeltia.com> wrote:
>>>>>>>>
>>>>>>>> Should I be seeing this in the log of a vanilla NiFi
>>>>>>>> installation on CentOS?
>>>>>>>>
>>>>>>>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to
>>>>>>>> java.util.ServiceConfigurationError:
>>>>>>>> org.apache.nifi.processor.Processor: Provider
>>>>>>>> org.apache.nifi.processors.hive.PutHiveStreaming could not be
>>>>>>>> initialized
>>>>>>>> .
>>>>>>>> .
>>>>>>>> .
>>>>>>>> Caused by: org.xerial.snappy.SnappyError:
>>>>>>>> [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>>>>>>>>
>>>>>>>> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down
>>>>>>>> the wrong /nifi-1.11.4-bin.zip/?
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>
Re: Hive NAR not loading because of snappy?
Posted by Matt Burgess <ma...@gmail.com>.
Understood. But outside of a permissions issue or usage of CompressContent or something else that would try to load the same version of Snappy, I’m not sure what’s going on there. AFAIK other folks install this on centos without issue. The related snappy issues were supposedly resolved well before 1.11.4. Did it used to work on previous versions? Have you tried the latest release to see if it works? Has the version of Java changed?
Sent from my iPhone
> On Oct 13, 2020, at 8:13 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>
> We didn't want to make deletion or modification of a shipped component to be a required part of its installation since we don't produce the NiFi download. We'd rather install it as it comes.
>
> On 10/13/20 4:45 PM, Matt Burgess wrote:
>> Ouch! It does happen on the loading of the NAR to ensure the native library gets loaded. If you are not using Hive I’d think you could safely delete the nifi-hive-nar and it shouldn’t happen. Hard to tell why the native library couldn’t be installed though.
>>
>>
>>> On Oct 13, 2020, at 6:26 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>>>
>>> No, we don't even use (nor have we ever used) Hive in our flows. It's just there and we didn't want to modify the NiFi download. Should this not even happen if we're not using it?
>>>
>>> On 10/13/20 4:24 PM, Matt Burgess wrote:
>>>> Ugh now I remember, that version of Hive uses a version of Snappy that doesn’t create a unique path under /tmp, do you have multiple PutHiveStreaming processors in the flow? I don’t think that works because we can’t load a single native library into multiple classloaders.
>>>>
>>>>
>>>>> On Oct 13, 2020, at 6:15 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>>>>>
>>>>> I see
>>>>>
>>>>> -rwxr-xr-x. 1 nifi nifi 48432 Oct 13 13:48 snappy-1.0.5-libsnappyjava.so
>>>>>
>>>>> in /tmp. Therefore, not a permissions issue? Launching this way works:
>>>>>
>>>>> $ ( export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" && /opt/nifi/bin/nifi.sh start )
>>>>>
>>>>> but it's not something we want to do (in case that shared object disappears from /tmp).
>>>>>
>>>>>
>>>>> On 10/13/20 3:42 PM, Matt Burgess wrote:
>>>>>> IIRC this is likely a permissions issue, Xerial Snappy tries to unzip the native library to the location pointed to by “java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi user have write access to that directory? If not you can change the Java temp dir or set it specifically for Snappy (I don’t have the property on hand but a quick Google should find it)
>>>>>>
>>>>>> Regards,
>>>>>> Matt
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>>> On Oct 13, 2020, at 5:36 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>>>>>>>
>>>>>>> Should I be seeing this in the log of a vanilla NiFi installation on CentOS?
>>>>>>>
>>>>>>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized
>>>>>>> .
>>>>>>> .
>>>>>>> .
>>>>>>> Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>>>>>>>
>>>>>>> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the wrong nifi-1.11.4-bin.zip?
>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: Hive NAR not loading because of snappy?
Posted by Russell Bateman <ru...@windofkeltia.com>.
We didn't want to make deletion or modification of a shipped component
to be a required part of its installation since we don't produce the
NiFi download. We'd rather install it as it comes.
On 10/13/20 4:45 PM, Matt Burgess wrote:
> Ouch! It does happen on the loading of the NAR to ensure the native
> library gets loaded. If you are not using Hive I’d think you could
> safely delete the nifi-hive-nar and it shouldn’t happen. Hard to tell
> why the native library couldn’t be installed though.
>
>
>> On Oct 13, 2020, at 6:26 PM, Russell Bateman <ru...@windofkeltia.com>
>> wrote:
>>
>> No, we don't even use (nor have we ever used) Hive in our flows.
>> It's just there and we didn't want to modify the NiFi download.
>> Should this not even happen if we're not using it?
>>
>> On 10/13/20 4:24 PM, Matt Burgess wrote:
>>> Ugh now I remember, that version of Hive uses a version of Snappy
>>> that doesn’t create a unique path under /tmp, do you have multiple
>>> PutHiveStreaming processors in the flow? I don’t think that works
>>> because we can’t load a single native library into multiple
>>> classloaders.
>>>
>>>
>>>> On Oct 13, 2020, at 6:15 PM, Russell Bateman
>>>> <ru...@windofkeltia.com> wrote:
>>>>
>>>> I see
>>>>
>>>> -rwxr-xr-x. 1 nifi nifi 48432 Oct 13 13:48
>>>> snappy-1.0.5-libsnappyjava.so
>>>>
>>>> in //tmp/. Therefore, not a permissions issue? Launching this way
>>>> works:
>>>>
>>>> $ ( export
>>>> LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so"
>>>> && /opt/nifi/bin/nifi.sh start )
>>>>
>>>> but it's not something we want to do (in case that shared object
>>>> disappears from //tmp/).
>>>>
>>>>
>>>> On 10/13/20 3:42 PM, Matt Burgess wrote:
>>>>> IIRC this is likely a permissions issue, Xerial Snappy tries to
>>>>> unzip the native library to the location pointed to by
>>>>> “java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi
>>>>> user have write access to that directory? If not you can change
>>>>> the Java temp dir or set it specifically for Snappy (I don’t have
>>>>> the property on hand but a quick Google should find it)
>>>>>
>>>>> Regards,
>>>>> Matt
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On Oct 13, 2020, at 5:36 PM, Russell Bateman
>>>>>> <ru...@windofkeltia.com> wrote:
>>>>>>
>>>>>> Should I be seeing this in the log of a vanilla NiFi
>>>>>> installation on CentOS?
>>>>>>
>>>>>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to
>>>>>> java.util.ServiceConfigurationError:
>>>>>> org.apache.nifi.processor.Processor: Provider
>>>>>> org.apache.nifi.processors.hive.PutHiveStreaming could not be
>>>>>> initialized
>>>>>> .
>>>>>> .
>>>>>> .
>>>>>> Caused by: org.xerial.snappy.SnappyError:
>>>>>> [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>>>>>>
>>>>>> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down
>>>>>> the wrong /nifi-1.11.4-bin.zip/?
>>>>>>
>>>>>>
>>>>
>>
Re: Hive NAR not loading because of snappy?
Posted by Matt Burgess <ma...@gmail.com>.
Ouch! It does happen on the loading of the NAR to ensure the native library gets loaded. If you are not using Hive I’d think you could safely delete the nifi-hive-nar and it shouldn’t happen. Hard to tell why the native library couldn’t be installed though.
> On Oct 13, 2020, at 6:26 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>
> No, we don't even use (nor have we ever used) Hive in our flows. It's just there and we didn't want to modify the NiFi download. Should this not even happen if we're not using it?
>
>> On 10/13/20 4:24 PM, Matt Burgess wrote:
>> Ugh now I remember, that version of Hive uses a version of Snappy that doesn’t create a unique path under /tmp, do you have multiple PutHiveStreaming processors in the flow? I don’t think that works because we can’t load a single native library into multiple classloaders.
>>
>>
>>> On Oct 13, 2020, at 6:15 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>>>
>>> I see
>>>
>>> -rwxr-xr-x. 1 nifi nifi 48432 Oct 13 13:48 snappy-1.0.5-libsnappyjava.so
>>>
>>> in /tmp. Therefore, not a permissions issue? Launching this way works:
>>>
>>> $ ( export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" && /opt/nifi/bin/nifi.sh start )
>>>
>>> but it's not something we want to do (in case that shared object disappears from /tmp).
>>>
>>>
>>> On 10/13/20 3:42 PM, Matt Burgess wrote:
>>>> IIRC this is likely a permissions issue, Xerial Snappy tries to unzip the native library to the location pointed to by “java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi user have write access to that directory? If not you can change the Java temp dir or set it specifically for Snappy (I don’t have the property on hand but a quick Google should find it)
>>>>
>>>> Regards,
>>>> Matt
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On Oct 13, 2020, at 5:36 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>>>>>
>>>>> Should I be seeing this in the log of a vanilla NiFi installation on CentOS?
>>>>>
>>>>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized
>>>>> .
>>>>> .
>>>>> .
>>>>> Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>>>>>
>>>>> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the wrong nifi-1.11.4-bin.zip?
>>>>>
>>>>>
>>>
>
Re: Hive NAR not loading because of snappy?
Posted by Russell Bateman <ru...@windofkeltia.com>.
No, we don't even use (nor have we ever used) Hive in our flows. It's
just there and we didn't want to modify the NiFi download. Should this
not even happen if we're not using it?
On 10/13/20 4:24 PM, Matt Burgess wrote:
> Ugh now I remember, that version of Hive uses a version of Snappy that
> doesn’t create a unique path under /tmp, do you have multiple
> PutHiveStreaming processors in the flow? I don’t think that works
> because we can’t load a single native library into multiple classloaders.
>
>
>> On Oct 13, 2020, at 6:15 PM, Russell Bateman <ru...@windofkeltia.com>
>> wrote:
>>
>> I see
>>
>> -rwxr-xr-x. 1 nifi nifi 48432 Oct 13 13:48
>> snappy-1.0.5-libsnappyjava.so
>>
>> in //tmp/. Therefore, not a permissions issue? Launching this way works:
>>
>> $ ( export
>> LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so"
>> && /opt/nifi/bin/nifi.sh start )
>>
>> but it's not something we want to do (in case that shared object
>> disappears from //tmp/).
>>
>>
>> On 10/13/20 3:42 PM, Matt Burgess wrote:
>>> IIRC this is likely a permissions issue, Xerial Snappy tries to
>>> unzip the native library to the location pointed to by
>>> “java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi user
>>> have write access to that directory? If not you can change the Java
>>> temp dir or set it specifically for Snappy (I don’t have the
>>> property on hand but a quick Google should find it)
>>>
>>> Regards,
>>> Matt
>>>
>>> Sent from my iPhone
>>>
>>>> On Oct 13, 2020, at 5:36 PM, Russell Bateman
>>>> <ru...@windofkeltia.com> wrote:
>>>>
>>>> Should I be seeing this in the log of a vanilla NiFi installation
>>>> on CentOS?
>>>>
>>>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to
>>>> java.util.ServiceConfigurationError:
>>>> org.apache.nifi.processor.Processor: Provider
>>>> org.apache.nifi.processors.hive.PutHiveStreaming could not be
>>>> initialized
>>>> .
>>>> .
>>>> .
>>>> Caused by: org.xerial.snappy.SnappyError:
>>>> [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>>>>
>>>> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the
>>>> wrong /nifi-1.11.4-bin.zip/?
>>>>
>>>>
>>
Re: Hive NAR not loading because of snappy?
Posted by Matt Burgess <ma...@gmail.com>.
Ugh now I remember, that version of Hive uses a version of Snappy that doesn’t create a unique path under /tmp, do you have multiple PutHiveStreaming processors in the flow? I don’t think that works because we can’t load a single native library into multiple classloaders.
> On Oct 13, 2020, at 6:15 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>
> I see
>
> -rwxr-xr-x. 1 nifi nifi 48432 Oct 13 13:48 snappy-1.0.5-libsnappyjava.so
>
> in /tmp. Therefore, not a permissions issue? Launching this way works:
>
> $ ( export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" && /opt/nifi/bin/nifi.sh start )
>
> but it's not something we want to do (in case that shared object disappears from /tmp).
>
>
> On 10/13/20 3:42 PM, Matt Burgess wrote:
>> IIRC this is likely a permissions issue, Xerial Snappy tries to unzip the native library to the location pointed to by “java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi user have write access to that directory? If not you can change the Java temp dir or set it specifically for Snappy (I don’t have the property on hand but a quick Google should find it)
>>
>> Regards,
>> Matt
>>
>> Sent from my iPhone
>>
>>> On Oct 13, 2020, at 5:36 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>>>
>>> Should I be seeing this in the log of a vanilla NiFi installation on CentOS?
>>>
>>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized
>>> .
>>> .
>>> .
>>> Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>>>
>>> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the wrong nifi-1.11.4-bin.zip?
>>>
>>>
>
Re: Hive NAR not loading because of snappy?
Posted by Russell Bateman <ru...@windofkeltia.com>.
I see
-rwxr-xr-x. 1 nifi nifi 48432 Oct 13 13:48
snappy-1.0.5-libsnappyjava.so
in //tmp/. Therefore, not a permissions issue? Launching this way works:
$ ( export
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" &&
/opt/nifi/bin/nifi.sh start )
but it's not something we want to do (in case that shared object
disappears from //tmp/).
On 10/13/20 3:42 PM, Matt Burgess wrote:
> IIRC this is likely a permissions issue, Xerial Snappy tries to unzip
> the native library to the location pointed to by “java.io.tempdir”
> which on *nix defaults to /tmp. Does the NiFi user have write access
> to that directory? If not you can change the Java temp dir or set it
> specifically for Snappy (I don’t have the property on hand but a quick
> Google should find it)
>
> Regards,
> Matt
>
> Sent from my iPhone
>
>> On Oct 13, 2020, at 5:36 PM, Russell Bateman <ru...@windofkeltia.com>
>> wrote:
>>
>> Should I be seeing this in the log of a vanilla NiFi installation
>> on CentOS?
>>
>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to
>> java.util.ServiceConfigurationError:
>> org.apache.nifi.processor.Processor: Provider
>> org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized
>> .
>> .
>> .
>> Caused by: org.xerial.snappy.SnappyError:
>> [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>>
>> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the
>> wrong /nifi-1.11.4-bin.zip/?
>>
>>
Re: Hive NAR not loading because of snappy?
Posted by Matt Burgess <ma...@gmail.com>.
IIRC this is likely a permissions issue, Xerial Snappy tries to unzip the native library to the location pointed to by “java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi user have write access to that directory? If not you can change the Java temp dir or set it specifically for Snappy (I don’t have the property on hand but a quick Google should find it)
Regards,
Matt
Sent from my iPhone
> On Oct 13, 2020, at 5:36 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
>
> Should I be seeing this in the log of a vanilla NiFi installation on CentOS?
>
> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized
> .
> .
> .
> Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
>
> "Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the wrong nifi-1.11.4-bin.zip?
>
>