You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Deepesh Maheshwari <de...@gmail.com> on 2015/08/04 09:11:00 UTC
Unable to load native-hadoop library for your platform
Hi,
When i run the spark locally on windows it gives below hadoop library error.
I am using below spark version.
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.4.1</version>
</dependency>
2015-08-04 12:22:23,463 WARN (org.apache.hadoop.util.NativeCodeLoader:62)
- Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
Tried to find it on internet but not able to find exact root cause.
Please let me know what is it, why it is giving warning and how can i
resolve it.
Thanks,
Deepesh
Re: Unable to load native-hadoop library for your platform
Posted by Steve Loughran <st...@hortonworks.com>.
> On 4 Aug 2015, at 12:26, Sean Owen <so...@cloudera.com> wrote:
>
> Oh good point, does the Windows integration need native libs for
> POSIX-y file system access? I know there are some binaries shipped for
> this purpose but wasn't sure if that's part of what's covered in the
> native libs message.
On unix its a warning that you'll get slower access to bits of the system and no posix-ish support, on windows its more serious (and should be reported more seriously)
It's used for various low-level things, and currently Hadoop fails somewhat messily when its not there. That's even for things like MiniYarnCluster, which is a real annoyance.
I have a snapshot of the Hadoop 2.6 windows binaries if people want them; I should stick up a copy of the 2.7.1 ones too
https://github.com/steveloughran/clusterconfigs/tree/master/clusters/morzine/hadoop_home/
>
> On Tue, Aug 4, 2015 at 6:01 PM, Steve Loughran <st...@hortonworks.com> wrote:
>> Think it may be needed on Windows, certainly if you start trying to work with local files.
>>
>>
>>> On 4 Aug 2015, at 00:34, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> It won't affect you if you're not actually running Hadoop. But it's
>>> mainly things like Snappy/LZO compression which are implemented as
>>> native libraries under the hood.
>>
>> There's a lot more in those native libs, primarily to bypass bits missing from the java APIs (FS permissions) and to add new features (encryption, soon erasure coding).
>>
>> The Hadoop file:// FS uses it on windows, at least for now
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to load native-hadoop library for your platform
Posted by Sean Owen <so...@cloudera.com>.
Oh good point, does the Windows integration need native libs for
POSIX-y file system access? I know there are some binaries shipped for
this purpose but wasn't sure if that's part of what's covered in the
native libs message.
On Tue, Aug 4, 2015 at 6:01 PM, Steve Loughran <st...@hortonworks.com> wrote:
> Think it may be needed on Windows, certainly if you start trying to work with local files.
>
>
>> On 4 Aug 2015, at 00:34, Sean Owen <so...@cloudera.com> wrote:
>>
>> It won't affect you if you're not actually running Hadoop. But it's
>> mainly things like Snappy/LZO compression which are implemented as
>> native libraries under the hood.
>
> There's a lot more in those native libs, primarily to bypass bits missing from the java APIs (FS permissions) and to add new features (encryption, soon erasure coding).
>
> The Hadoop file:// FS uses it on windows, at least for now
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to load native-hadoop library for your platform
Posted by Steve Loughran <st...@hortonworks.com>.
Think it may be needed on Windows, certainly if you start trying to work with local files.
> On 4 Aug 2015, at 00:34, Sean Owen <so...@cloudera.com> wrote:
>
> It won't affect you if you're not actually running Hadoop. But it's
> mainly things like Snappy/LZO compression which are implemented as
> native libraries under the hood.
There's a lot more in those native libs, primarily to bypass bits missing from the java APIs (FS permissions) and to add new features (encryption, soon erasure coding).
The Hadoop file:// FS uses it on windows, at least for now
> Spark doesn't necessarily use these
> anyway; it's from the Hadoop libs.
>
> On Tue, Aug 4, 2015 at 8:30 AM, Deepesh Maheshwari
> <de...@gmail.com> wrote:
>> Can you elaborate about the things this native library covering.
>> One you mentioned accelerated compression.
>>
>> It would be very helpful if you can give any useful to link to read more
>> about it.
>>
>> On Tue, Aug 4, 2015 at 12:56 PM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> You can ignore it entirely. It just means you haven't installed and
>>> configured native libraries for things like accelerated compression,
>>> but it has no negative impact otherwise.
>>>
>>> On Tue, Aug 4, 2015 at 8:11 AM, Deepesh Maheshwari
>>> <de...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> When i run the spark locally on windows it gives below hadoop library
>>>> error.
>>>> I am using below spark version.
>>>>
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-core_2.10</artifactId>
>>>> <version>1.4.1</version>
>>>> </dependency>
>>>>
>>>>
>>>> 2015-08-04 12:22:23,463 WARN
>>>> (org.apache.hadoop.util.NativeCodeLoader:62) -
>>>> Unable to load native-hadoop library for your platform... using
>>>> builtin-java
>>>> classes where applicable
>>>>
>>>> Tried to find it on internet but not able to find exact root cause.
>>>> Please let me know what is it, why it is giving warning and how can i
>>>> resolve it.
>>>>
>>>> Thanks,
>>>> Deepesh
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to load native-hadoop library for your platform
Posted by Deepesh Maheshwari <de...@gmail.com>.
Hi Sean,
Thanks for the information and clarification.
On Tue, Aug 4, 2015 at 1:04 PM, Sean Owen <so...@cloudera.com> wrote:
> It won't affect you if you're not actually running Hadoop. But it's
> mainly things like Snappy/LZO compression which are implemented as
> native libraries under the hood. Spark doesn't necessarily use these
> anyway; it's from the Hadoop libs.
>
> On Tue, Aug 4, 2015 at 8:30 AM, Deepesh Maheshwari
> <de...@gmail.com> wrote:
> > Can you elaborate about the things this native library covering.
> > One you mentioned accelerated compression.
> >
> > It would be very helpful if you can give any useful to link to read more
> > about it.
> >
> > On Tue, Aug 4, 2015 at 12:56 PM, Sean Owen <so...@cloudera.com> wrote:
> >>
> >> You can ignore it entirely. It just means you haven't installed and
> >> configured native libraries for things like accelerated compression,
> >> but it has no negative impact otherwise.
> >>
> >> On Tue, Aug 4, 2015 at 8:11 AM, Deepesh Maheshwari
> >> <de...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > When i run the spark locally on windows it gives below hadoop library
> >> > error.
> >> > I am using below spark version.
> >> >
> >> > <dependency>
> >> > <groupId>org.apache.spark</groupId>
> >> > <artifactId>spark-core_2.10</artifactId>
> >> > <version>1.4.1</version>
> >> > </dependency>
> >> >
> >> >
> >> > 2015-08-04 12:22:23,463 WARN
> >> > (org.apache.hadoop.util.NativeCodeLoader:62) -
> >> > Unable to load native-hadoop library for your platform... using
> >> > builtin-java
> >> > classes where applicable
> >> >
> >> > Tried to find it on internet but not able to find exact root cause.
> >> > Please let me know what is it, why it is giving warning and how can i
> >> > resolve it.
> >> >
> >> > Thanks,
> >> > Deepesh
> >
> >
>
Re: Unable to load native-hadoop library for your platform
Posted by Sean Owen <so...@cloudera.com>.
It won't affect you if you're not actually running Hadoop. But it's
mainly things like Snappy/LZO compression which are implemented as
native libraries under the hood. Spark doesn't necessarily use these
anyway; it's from the Hadoop libs.
On Tue, Aug 4, 2015 at 8:30 AM, Deepesh Maheshwari
<de...@gmail.com> wrote:
> Can you elaborate about the things this native library covering.
> One you mentioned accelerated compression.
>
> It would be very helpful if you can give any useful to link to read more
> about it.
>
> On Tue, Aug 4, 2015 at 12:56 PM, Sean Owen <so...@cloudera.com> wrote:
>>
>> You can ignore it entirely. It just means you haven't installed and
>> configured native libraries for things like accelerated compression,
>> but it has no negative impact otherwise.
>>
>> On Tue, Aug 4, 2015 at 8:11 AM, Deepesh Maheshwari
>> <de...@gmail.com> wrote:
>> > Hi,
>> >
>> > When i run the spark locally on windows it gives below hadoop library
>> > error.
>> > I am using below spark version.
>> >
>> > <dependency>
>> > <groupId>org.apache.spark</groupId>
>> > <artifactId>spark-core_2.10</artifactId>
>> > <version>1.4.1</version>
>> > </dependency>
>> >
>> >
>> > 2015-08-04 12:22:23,463 WARN
>> > (org.apache.hadoop.util.NativeCodeLoader:62) -
>> > Unable to load native-hadoop library for your platform... using
>> > builtin-java
>> > classes where applicable
>> >
>> > Tried to find it on internet but not able to find exact root cause.
>> > Please let me know what is it, why it is giving warning and how can i
>> > resolve it.
>> >
>> > Thanks,
>> > Deepesh
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Unable to load native-hadoop library for your platform
Posted by Deepesh Maheshwari <de...@gmail.com>.
Can you elaborate about the things this native library covering.
One you mentioned accelerated compression.
It would be very helpful if you can give any useful to link to read more
about it.
On Tue, Aug 4, 2015 at 12:56 PM, Sean Owen <so...@cloudera.com> wrote:
> You can ignore it entirely. It just means you haven't installed and
> configured native libraries for things like accelerated compression,
> but it has no negative impact otherwise.
>
> On Tue, Aug 4, 2015 at 8:11 AM, Deepesh Maheshwari
> <de...@gmail.com> wrote:
> > Hi,
> >
> > When i run the spark locally on windows it gives below hadoop library
> error.
> > I am using below spark version.
> >
> > <dependency>
> > <groupId>org.apache.spark</groupId>
> > <artifactId>spark-core_2.10</artifactId>
> > <version>1.4.1</version>
> > </dependency>
> >
> >
> > 2015-08-04 12:22:23,463 WARN
> (org.apache.hadoop.util.NativeCodeLoader:62) -
> > Unable to load native-hadoop library for your platform... using
> builtin-java
> > classes where applicable
> >
> > Tried to find it on internet but not able to find exact root cause.
> > Please let me know what is it, why it is giving warning and how can i
> > resolve it.
> >
> > Thanks,
> > Deepesh
>
Re: Unable to load native-hadoop library for your platform
Posted by Sean Owen <so...@cloudera.com>.
You can ignore it entirely. It just means you haven't installed and
configured native libraries for things like accelerated compression,
but it has no negative impact otherwise.
On Tue, Aug 4, 2015 at 8:11 AM, Deepesh Maheshwari
<de...@gmail.com> wrote:
> Hi,
>
> When i run the spark locally on windows it gives below hadoop library error.
> I am using below spark version.
>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.10</artifactId>
> <version>1.4.1</version>
> </dependency>
>
>
> 2015-08-04 12:22:23,463 WARN (org.apache.hadoop.util.NativeCodeLoader:62) -
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> Tried to find it on internet but not able to find exact root cause.
> Please let me know what is it, why it is giving warning and how can i
> resolve it.
>
> Thanks,
> Deepesh
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org