You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by ro...@hyatt.com on 2016/10/19 11:07:17 UTC
native snappy library not available
I have flink running on a standalone cluster, but with a Hadoop cluster available to it. I’m using a RollingSink and a SequenceFileWriter, and I’ve set compression to Snappy:
.setWriter(new SequenceFileWriter<Text, Text>("org.apache.hadoop.io.compress.SnappyCodec", SequenceFile.CompressionType.BLOCK))
However, when I try to run this job, I get an error indicating that the native snappy ibrary is not available:
Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
I’ve tried adding the snappy java jar to the classpath. I’ve also tried adding the native dir to the classpath. Neither have helped.
-C file://usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar
-C file://usr/hdp/current/hadoop/lib/native/
Any thoughts on how I can get this to work?
Thanks!
________________________________
The information contained in this communication is confidential and intended only for the use of the recipient named above, and may be legally privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend it to the sender and delete the original message and copy of it from your computer system. Opinions, conclusions and other information in this message that do not relate to our official business should be understood as neither given nor endorsed by the company.
Re: native snappy library not available
Posted by ro...@hyatt.com.
Hi Max,
Thanks for the feedback. The SnappyCodec is specified in io.compression.codecs. But, for whatever reason, the issue appears to be that Flink is unable to pick up the native Snappy libraries. Snappy compression works fine from w/in the Hadoop cluster (e.g. using a Hive query), but not for Flink (running on a standalone cluster) when attempting to write to HDFS. We are likely changing our cluster to be YARN managed in the next week. I don’t know if that will change anything, but we’ll test again after to see.
Incidentally, using org.apache.hadoop.io.compress.DefaultCodec works fine. I haven’t tried other compression codecs, yet.
On 10/19/16, 6:33 PM, "Maximilian Michels" <mx...@apache.org> wrote:
The Hadoop config of your Hadoop installation which is loaded in
SequenceFileWriter.open() needs to be configured to have
"io.compression.codecs" set to include "SnappyCodec". This is probably
described in the Hadoop documentation.
-Max
On Wed, Oct 19, 2016 at 6:09 PM, <ro...@hyatt.com> wrote:
> Hi Till,
>
>
>
> Thanks for the response. I’m hoping not to have to change Flink’s lib
> folder. The path I specified does exist on each node, and –C is supposed
> to add the path to the classpath on all nodes in the cluster. I might try
> to bundle the snappy jar within my job jar to see if that works.
>
>
>
> From: Till Rohrmann <tr...@apache.org>
> Reply-To: "user@flink.apache.org" <us...@flink.apache.org>
> Date: Wednesday, October 19, 2016 at 2:46 PM
> To: "user@flink.apache.org" <us...@flink.apache.org>
> Subject: Re: native snappy library not available
>
>
>
> Hi Robert,
>
>
>
> have you tried putting the snappy java jar in Flink's lib folder? When
> specifying the classpath manually you have to make sure that all distributed
> components are also started with this classpath.
>
>
>
> Cheers,
>
> Till
>
>
>
> On Wed, Oct 19, 2016 at 1:07 PM, <ro...@hyatt.com> wrote:
>
> I have flink running on a standalone cluster, but with a Hadoop cluster
> available to it. I’m using a RollingSink and a SequenceFileWriter, and I’ve
> set compression to Snappy:
>
>
>
> .setWriter(new SequenceFileWriter<Text,
> Text>("org.apache.hadoop.io.compress.SnappyCodec",
> SequenceFile.CompressionType.BLOCK))
>
>
>
>
>
> However, when I try to run this job, I get an error indicating that the
> native snappy ibrary is not available:
>
>
>
> Caused by: java.lang.RuntimeException: native snappy library not available:
> this version of libhadoop was built without snappy support.
>
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
>
>
>
> I’ve tried adding the snappy java jar to the classpath. I’ve also tried
> adding the native dir to the classpath. Neither have helped.
>
> -C file://usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar
>
> -C file://usr/hdp/current/hadoop/lib/native/
>
>
>
> Any thoughts on how I can get this to work?
>
>
>
> Thanks!
>
>
>
>
>
> ________________________________
>
> The information contained in this communication is confidential and intended
> only for the use of the recipient named above, and may be legally privileged
> and exempt from disclosure under applicable law. If the reader of this
> message is not the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please resend
> it to the sender and delete the original message and copy of it from your
> computer system. Opinions, conclusions and other information in this message
> that do not relate to our official business should be understood as neither
> given nor endorsed by the company.
>
>
>
>
> ________________________________
> The information contained in this communication is confidential and intended
> only for the use of the recipient named above, and may be legally privileged
> and exempt from disclosure under applicable law. If the reader of this
> message is not the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please resend
> it to the sender and delete the original message and copy of it from your
> computer system. Opinions, conclusions and other information in this message
> that do not relate to our official business should be understood as neither
> given nor endorsed by the company.
________________________________
The information contained in this communication is confidential and intended only for the use of the recipient named above, and may be legally privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend it to the sender and delete the original message and copy of it from your computer system. Opinions, conclusions and other information in this message that do not relate to our official business should be understood as neither given nor endorsed by the company.
Re: native snappy library not available
Posted by Maximilian Michels <mx...@apache.org>.
The Hadoop config of your Hadoop installation which is loaded in
SequenceFileWriter.open() needs to be configured to have
"io.compression.codecs" set to include "SnappyCodec". This is probably
described in the Hadoop documentation.
-Max
On Wed, Oct 19, 2016 at 6:09 PM, <ro...@hyatt.com> wrote:
> Hi Till,
>
>
>
> Thanks for the response. I’m hoping not to have to change Flink’s lib
> folder. The path I specified does exist on each node, and –C is supposed
> to add the path to the classpath on all nodes in the cluster. I might try
> to bundle the snappy jar within my job jar to see if that works.
>
>
>
> From: Till Rohrmann <tr...@apache.org>
> Reply-To: "user@flink.apache.org" <us...@flink.apache.org>
> Date: Wednesday, October 19, 2016 at 2:46 PM
> To: "user@flink.apache.org" <us...@flink.apache.org>
> Subject: Re: native snappy library not available
>
>
>
> Hi Robert,
>
>
>
> have you tried putting the snappy java jar in Flink's lib folder? When
> specifying the classpath manually you have to make sure that all distributed
> components are also started with this classpath.
>
>
>
> Cheers,
>
> Till
>
>
>
> On Wed, Oct 19, 2016 at 1:07 PM, <ro...@hyatt.com> wrote:
>
> I have flink running on a standalone cluster, but with a Hadoop cluster
> available to it. I’m using a RollingSink and a SequenceFileWriter, and I’ve
> set compression to Snappy:
>
>
>
> .setWriter(new SequenceFileWriter<Text,
> Text>("org.apache.hadoop.io.compress.SnappyCodec",
> SequenceFile.CompressionType.BLOCK))
>
>
>
>
>
> However, when I try to run this job, I get an error indicating that the
> native snappy ibrary is not available:
>
>
>
> Caused by: java.lang.RuntimeException: native snappy library not available:
> this version of libhadoop was built without snappy support.
>
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
>
>
>
> I’ve tried adding the snappy java jar to the classpath. I’ve also tried
> adding the native dir to the classpath. Neither have helped.
>
> -C file://usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar
>
> -C file://usr/hdp/current/hadoop/lib/native/
>
>
>
> Any thoughts on how I can get this to work?
>
>
>
> Thanks!
>
>
>
>
>
> ________________________________
>
> The information contained in this communication is confidential and intended
> only for the use of the recipient named above, and may be legally privileged
> and exempt from disclosure under applicable law. If the reader of this
> message is not the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please resend
> it to the sender and delete the original message and copy of it from your
> computer system. Opinions, conclusions and other information in this message
> that do not relate to our official business should be understood as neither
> given nor endorsed by the company.
>
>
>
>
> ________________________________
> The information contained in this communication is confidential and intended
> only for the use of the recipient named above, and may be legally privileged
> and exempt from disclosure under applicable law. If the reader of this
> message is not the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please resend
> it to the sender and delete the original message and copy of it from your
> computer system. Opinions, conclusions and other information in this message
> that do not relate to our official business should be understood as neither
> given nor endorsed by the company.
Re: native snappy library not available
Posted by ro...@hyatt.com.
Hi Till,
Thanks for the response. I’m hoping not to have to change Flink’s lib folder. The path I specified does exist on each node, and –C is supposed to add the path to the classpath on all nodes in the cluster. I might try to bundle the snappy jar within my job jar to see if that works.
From: Till Rohrmann <tr...@apache.org>
Reply-To: "user@flink.apache.org" <us...@flink.apache.org>
Date: Wednesday, October 19, 2016 at 2:46 PM
To: "user@flink.apache.org" <us...@flink.apache.org>
Subject: Re: native snappy library not available
Hi Robert,
have you tried putting the snappy java jar in Flink's lib folder? When specifying the classpath manually you have to make sure that all distributed components are also started with this classpath.
Cheers,
Till
On Wed, Oct 19, 2016 at 1:07 PM, <ro...@hyatt.com>> wrote:
I have flink running on a standalone cluster, but with a Hadoop cluster available to it. I’m using a RollingSink and a SequenceFileWriter, and I’ve set compression to Snappy:
.setWriter(new SequenceFileWriter<Text, Text>("org.apache.hadoop.io<http://org.apache.hadoop.io>.compress.SnappyCodec", SequenceFile.CompressionType.BLOCK))
However, when I try to run this job, I get an error indicating that the native snappy ibrary is not available:
Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
I’ve tried adding the snappy java jar to the classpath. I’ve also tried adding the native dir to the classpath. Neither have helped.
-C file://usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar
-C file://usr/hdp/current/hadoop/lib/native/
Any thoughts on how I can get this to work?
Thanks!
________________________________
The information contained in this communication is confidential and intended only for the use of the recipient named above, and may be legally privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend it to the sender and delete the original message and copy of it from your computer system. Opinions, conclusions and other information in this message that do not relate to our official business should be understood as neither given nor endorsed by the company.
________________________________
The information contained in this communication is confidential and intended only for the use of the recipient named above, and may be legally privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend it to the sender and delete the original message and copy of it from your computer system. Opinions, conclusions and other information in this message that do not relate to our official business should be understood as neither given nor endorsed by the company.
Re: native snappy library not available
Posted by Till Rohrmann <tr...@apache.org>.
Hi Robert,
have you tried putting the snappy java jar in Flink's lib folder? When
specifying the classpath manually you have to make sure that all
distributed components are also started with this classpath.
Cheers,
Till
On Wed, Oct 19, 2016 at 1:07 PM, <ro...@hyatt.com> wrote:
> I have flink running on a standalone cluster, but with a Hadoop cluster
> available to it. I’m using a RollingSink and a SequenceFileWriter, and
> I’ve set compression to Snappy:
>
>
>
> .setWriter(new SequenceFileWriter<Text, Text>("org.apache.hadoop.io.compress.SnappyCodec",
> SequenceFile.CompressionType.*BLOCK*))
>
>
>
>
>
> However, when I try to run this job, I get an error indicating that the
> native snappy ibrary is not available:
>
>
>
> Caused by: java.lang.RuntimeException: native snappy library not
> available: this version of libhadoop was built without snappy support.
>
> at org.apache.hadoop.io.compress.SnappyCodec.
> checkNativeCodeLoaded(SnappyCodec.java:65)
>
>
>
> I’ve tried adding the snappy java jar to the classpath. I’ve also tried
> adding the native dir to the classpath. Neither have helped.
>
> -C file://usr/hdp/current/hadoop-client/lib/snappy-java-1.0.4.1.jar
>
> -C file://usr/hdp/current/hadoop/lib/native/
>
>
>
> Any thoughts on how I can get this to work?
>
>
>
> Thanks!
>
>
>
> ------------------------------
> The information contained in this communication is confidential and
> intended only for the use of the recipient named above, and may be legally
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any dissemination, distribution or copying of this communication is
> strictly prohibited. If you have received this communication in error,
> please resend it to the sender and delete the original message and copy of
> it from your computer system. Opinions, conclusions and other information
> in this message that do not relate to our official business should be
> understood as neither given nor endorsed by the company.
>