You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Kashmar, Ali" <Al...@emc.com> on 2016/01/04 15:03:29 UTC
Re: Error due to Hadoop version mismatch
Hi Max,
Both commands return nothing. Those variables aren’t set.
The only software I installed on these machines is Flink and Java.
-Ali
On 2015-12-28, 6:42 AM, "Maximilian Michels" <mx...@apache.org> wrote:
>Hi Ali,
>
>The warning about the native Hadoop libraries is nothing to worry
>about. The native modules are platform-optimized modules which may be
>used to improve performance. They are not necessary for Hadoop to
>function correctly.
>
>The exception message implies that you are using is a very old version
>of Hadoop. Do you have other Hadoop versions installed on the same
>machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0
>without any problems.
>
>On the cluster machines, what is the output of these commands?
>
>echo $HADOOP_CLASSPATH
>echo $HADOOP_CONF_DIR
>
>
>Thanks,
>Max
>
>On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <Al...@emc.com> wrote:
>> Hi Max,
>>
>> I have the same output for the Task Manager:
>>
>> 11:25:04,274 INFO org.apache.flink.runtime.taskmanager.TaskManager
>> - Hadoop version: 2.6.0
>>
>> I do get this line at the beginning of both job and task manager log
>>files:
>>
>> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader
>> - Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>>
>> Do you think it has anything to do with it?
>>
>> Thanks,
>> Ali
>>
>> On 2015-12-23, 7:30 AM, "Maximilian Michels" <mx...@apache.org> wrote:
>>
>>>Hi Ali,
>>>
>>>Could you please also post the Hadoop version output of the task
>>>manager log files? It looks like the task managers are running a
>>>different Hadoop version.
>>>
>>>Thanks,
>>>Max
>>>
>>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali <Al...@emc.com>
>>>wrote:
>>>> Hi Robert,
>>>>
>>>> I found the version in the job manager log file:
>>>>
>>>> 17:33:49,636 INFO org.apache.flink.runtime.jobmanager.JobManager
>>>> - Hadoop version: 2.6.0
>>>>
>>>> But the Hadoop installation I have is saying this:
>>>>
>>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version
>>>> Hadoop 2.6.0
>>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
>>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
>>>> Compiled by jenkins on 2014-11-13T21:10Z
>>>> Compiled with protoc 2.5.0
>>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a
>>>> This command was run using
>>>>
>>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2
>>>>.6
>>>>.0
>>>> .jar
>>>>
>>>>
>>>> So one of them is lying to me? :)
>>>>
>>>> Ali
>>>>
>>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rm...@apache.org> wrote:
>>>>
>>>>>Hi Ali,
>>>>>
>>>>>the TaskManagers and the JobManager is logging the Hadoop version on
>>>>>startup.
>>>>>
>>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali <Al...@emc.com>
>>>>>wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I
>>>>>>downloaded
>>>>>>the
>>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I also
>>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and set
>>>>>>up
>>>>>>HDFS
>>>>>> on a separate machine. When I start Flink I get the following error:
>>>>>>
>>>>>> 17:34:13,047 INFO org.apache.flink.runtime.jobmanager.JobManager
>>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72 (Protocol
>>>>>>Event
>>>>>> Processing) changed to FAILED.
>>>>>> java.lang.Exception: Call to registerInputOutput() of invokable
>>>>>>failed
>>>>>> at
>>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Caused by: java.io.IOException: The given file URI (hdfs://
>>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS
>>>>>>NameNode
>>>>>>at
>>>>>> 10.13.182.171:9000, but the File System could not be initialized
>>>>>>with
>>>>>> that address: Server IPC version 9 cannot communicate with client
>>>>>>version 4
>>>>>> at
>>>>>>
>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFi
>>>>>>le
>>>>>>Sy
>>>>>>stem.java:337)
>>>>>> at
>>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253)
>>>>>> at
>>>>>>
>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsSta
>>>>>>te
>>>>>>Ba
>>>>>>ckend.java:142)
>>>>>> at
>>>>>>
>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsSta
>>>>>>te
>>>>>>Ba
>>>>>>ckend.java:101)
>>>>>> at
>>>>>>
>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.creat
>>>>>>eF
>>>>>>ro
>>>>>>mConfig(FsStateBackendFactory.java:48)
>>>>>> at
>>>>>>
>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBacken
>>>>>>d(
>>>>>>St
>>>>>>reamTask.java:517)
>>>>>> at
>>>>>>
>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutp
>>>>>>ut
>>>>>>(S
>>>>>>treamTask.java:171)
>>>>>> at
>>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>>>>>> ... 1 more
>>>>>> Caused by:
>>>>>>
>>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$Versi
>>>>>>on
>>>>>>Mi
>>>>>>smatch):
>>>>>> Server IPC version 9 cannot communicate with client version 4
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113)
>>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>Method)
>>>>>> at
>>>>>>
>>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>>>>>ja
>>>>>>va
>>>>>>:62)
>>>>>> at
>>>>>>
>>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>>>>>so
>>>>>>rI
>>>>>>mpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>> at
>>>>>>
>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
>>>>>>nv
>>>>>>oc
>>>>>>ationHandler.java:85)
>>>>>> at
>>>>>>
>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
>>>>>>io
>>>>>>nH
>>>>>>andler.java:62)
>>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source)
>>>>>> at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
>>>>>> at
>>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
>>>>>> at
>>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
>>>>>> at
>>>>>>
>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFi
>>>>>>le
>>>>>>Sy
>>>>>>stem.java:100)
>>>>>> at
>>>>>>
>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFi
>>>>>>le
>>>>>>Sy
>>>>>>stem.java:321)
>>>>>> ... 8 more
>>>>>>
>>>>>> I searched for this error online and it indicates that the client
>>>>>>which
>>>>>>is
>>>>>> Flink in this case is at a much lower version. Is there a way to
>>>>>>check
>>>>>>the
>>>>>> version of Hadoop packaged with my Flink installation?
>>>>>>
>>>>>> Thanks,
>>>>>> Ali
>>>>>>
>>>>
>>
Re: Error due to Hadoop version mismatch
Posted by "Kashmar, Ali" <Al...@emc.com>.
Looking at the lib folder revealed the problem. The lib folder on one of
the nodes had libraries for both hadoop 1 and 2. I’m not sure how I ended
up with that but it must have happened while I was copying the dependency
jars to each node. I removed all the jars and started with a fresh copy
and reran the test. It worked this time.
Thank you so much for being patient with me and sorry if I wasted your
time.
-Ali
On 2016-01-04, 3:17 PM, "Robert Metzger" <rm...@apache.org> wrote:
>Okay, then, my previous statement is false. From the stack trace, it seems
>that Flink is using an older Hadoop version. The DFSClient and RPC classes
>look different in Hadoop 2.6.0.
>Max checked already for some environment variables. Is $CLASSPATH set? Did
>you install Hadoop only by downloading the binaries or did you use
>something like the ubuntu package manager? (those packages sometimes place
>jar files in some lib/ folders)
>Did you put something additionally into the lib/ folder of Flink?
>
>I think when Flink is starting up, its also logging the classpath of the
>JVM. Can you post it here? (Maybe you need to start with DEBUG log level)
>
>
>On Mon, Jan 4, 2016 at 8:46 PM, Kashmar, Ali <Al...@emc.com> wrote:
>
>> Hi Robert,
>>
>> On the ubuntu host, the port 9000 is up:
>>
>> ubuntu@ubuntu-171:~$ netstat -tulpn
>> ....
>> tcp 0 0 10.13.182.171:9000 0.0.0.0:*
>>LISTEN
>> 8849/java
>> ....
>>
>> And the process using this port is:
>>
>> ubuntu@ubuntu-171:~$ ps -ef | grep 8849
>> ubuntu 8849 1 0 2015 ? 00:25:00
>> /usr/lib/jvm/java-8-oracle/bin/java -Dproc_namenode -Xmx1000m
>> -Djava.net.preferIPv4Stack=true
>> -Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
>> -Dhadoop.log.file=hadoop.log
>> -Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
>> -Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,console
>> -Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
>> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
>> -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
>> -Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
>> -Dhadoop.log.file=hadoop-ubuntu-namenode-ubuntu-171.log
>> -Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
>> -Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,RFA
>> -Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
>> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
>> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
>> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
>> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
>> -Dhadoop.security.logger=INFO,RFAS
>> org.apache.hadoop.hdfs.server.namenode.NameNode
>>
>> And the hadoop version is:
>>
>> ubuntu@ubuntu-171:~$ Documents/hadoop-2.6.0/bin/hadoop version
>> Hadoop 2.6.0
>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
>> Compiled by jenkins on 2014-11-13T21:10Z
>> Compiled with protoc 2.5.0
>> From source with checksum 18e43357c8f927c0695f1e9522859d6a
>> This command was run using
>>
>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6
>>.0
>> .jar
>>
>>
>>
>> -Ali
>>
>>
>>
>> On 2016-01-04, 2:20 PM, "Robert Metzger" <rm...@apache.org> wrote:
>>
>> >Your Flink installation has Hadoop 2.6.0 included, on the other
>>machine,
>> >there is a Hadoop version installed, which is most likely a 1.x or
>>even a
>> >0.x version.
>> >Are you sure that the host "ubuntu-171" has the ip 10.13.182.171 and
>>that
>> >the hadoop installation in the "/home/ubuntu/Documents/hadoop-2.6.0/"
>> >directory is listening on port 9000 ?
>> >
>> >On Mon, Jan 4, 2016 at 3:03 PM, Kashmar, Ali <Al...@emc.com>
>>wrote:
>> >
>> >> Hi Max,
>> >>
>> >> Both commands return nothing. Those variables aren’t set.
>> >>
>> >> The only software I installed on these machines is Flink and Java.
>> >>
>> >> -Ali
>> >>
>> >> On 2015-12-28, 6:42 AM, "Maximilian Michels" <mx...@apache.org> wrote:
>> >>
>> >> >Hi Ali,
>> >> >
>> >> >The warning about the native Hadoop libraries is nothing to worry
>> >> >about. The native modules are platform-optimized modules which may
>>be
>> >> >used to improve performance. They are not necessary for Hadoop to
>> >> >function correctly.
>> >> >
>> >> >The exception message implies that you are using is a very old
>>version
>> >> >of Hadoop. Do you have other Hadoop versions installed on the same
>> >> >machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0
>> >> >without any problems.
>> >> >
>> >> >On the cluster machines, what is the output of these commands?
>> >> >
>> >> >echo $HADOOP_CLASSPATH
>> >> >echo $HADOOP_CONF_DIR
>> >> >
>> >> >
>> >> >Thanks,
>> >> >Max
>> >> >
>> >> >On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <Al...@emc.com>
>> >> wrote:
>> >> >> Hi Max,
>> >> >>
>> >> >> I have the same output for the Task Manager:
>> >> >>
>> >> >> 11:25:04,274 INFO
>>org.apache.flink.runtime.taskmanager.TaskManager
>> >> >> - Hadoop version: 2.6.0
>> >> >>
>> >> >> I do get this line at the beginning of both job and task manager
>>log
>> >> >>files:
>> >> >>
>> >> >> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader
>> >> >> - Unable to load native-hadoop library for your platform...
>> >>using
>> >> >> builtin-java classes where applicable
>> >> >>
>> >> >> Do you think it has anything to do with it?
>> >> >>
>> >> >> Thanks,
>> >> >> Ali
>> >> >>
>> >> >> On 2015-12-23, 7:30 AM, "Maximilian Michels" <mx...@apache.org>
>>wrote:
>> >> >>
>> >> >>>Hi Ali,
>> >> >>>
>> >> >>>Could you please also post the Hadoop version output of the task
>> >> >>>manager log files? It looks like the task managers are running a
>> >> >>>different Hadoop version.
>> >> >>>
>> >> >>>Thanks,
>> >> >>>Max
>> >> >>>
>> >> >>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali
>><Al...@emc.com>
>> >> >>>wrote:
>> >> >>>> Hi Robert,
>> >> >>>>
>> >> >>>> I found the version in the job manager log file:
>> >> >>>>
>> >> >>>> 17:33:49,636 INFO
>>org.apache.flink.runtime.jobmanager.JobManager
>> >> >>>> - Hadoop version: 2.6.0
>> >> >>>>
>> >> >>>> But the Hadoop installation I have is saying this:
>> >> >>>>
>> >> >>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version
>> >> >>>> Hadoop 2.6.0
>> >> >>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
>> >> >>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
>> >> >>>> Compiled by jenkins on 2014-11-13T21:10Z
>> >> >>>> Compiled with protoc 2.5.0
>> >> >>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a
>> >> >>>> This command was run using
>> >> >>>>
>> >>
>>
>>>>>>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-comm
>>>>>>>>on
>> >>>>>>-2
>> >> >>>>.6
>> >> >>>>.0
>> >> >>>> .jar
>> >> >>>>
>> >> >>>>
>> >> >>>> So one of them is lying to me? :)
>> >> >>>>
>> >> >>>> Ali
>> >> >>>>
>> >> >>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rm...@apache.org>
>> >> wrote:
>> >> >>>>
>> >> >>>>>Hi Ali,
>> >> >>>>>
>> >> >>>>>the TaskManagers and the JobManager is logging the Hadoop
>>version
>> >>on
>> >> >>>>>startup.
>> >> >>>>>
>> >> >>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali
>><Ali.Kashmar@emc.com
>> >
>> >> >>>>>wrote:
>> >> >>>>>
>> >> >>>>>> Hello,
>> >> >>>>>>
>> >> >>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I
>> >> >>>>>>downloaded
>> >> >>>>>>the
>> >> >>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I
>>also
>> >> >>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and
>> >>set
>> >> >>>>>>up
>> >> >>>>>>HDFS
>> >> >>>>>> on a separate machine. When I start Flink I get the following
>> >>error:
>> >> >>>>>>
>> >> >>>>>> 17:34:13,047 INFO
>>org.apache.flink.runtime.jobmanager.JobManager
>> >> >>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72
>>(Protocol
>> >> >>>>>>Event
>> >> >>>>>> Processing) changed to FAILED.
>> >> >>>>>> java.lang.Exception: Call to registerInputOutput() of
>>invokable
>> >> >>>>>>failed
>> >> >>>>>> at
>> >> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
>> >> >>>>>> at java.lang.Thread.run(Thread.java:745)
>> >> >>>>>> Caused by: java.io.IOException: The given file URI (hdfs://
>> >> >>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS
>> >> >>>>>>NameNode
>> >> >>>>>>at
>> >> >>>>>> 10.13.182.171:9000, but the File System could not be
>>initialized
>> >> >>>>>>with
>> >> >>>>>> that address: Server IPC version 9 cannot communicate with
>>client
>> >> >>>>>>version 4
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hado
>>>>>>>>>>op
>> >>>>>>>>Fi
>> >> >>>>>>le
>> >> >>>>>>Sy
>> >> >>>>>>stem.java:337)
>> >> >>>>>> at
>> >> >>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(F
>>>>>>>>>>sS
>> >>>>>>>>ta
>> >> >>>>>>te
>> >> >>>>>>Ba
>> >> >>>>>>ckend.java:142)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(F
>>>>>>>>>>sS
>> >>>>>>>>ta
>> >> >>>>>>te
>> >> >>>>>>Ba
>> >> >>>>>>ckend.java:101)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.c
>>>>>>>>>>re
>> >>>>>>>>at
>> >> >>>>>>eF
>> >> >>>>>>ro
>> >> >>>>>>mConfig(FsStateBackendFactory.java:48)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBa
>>>>>>>>>>ck
>> >>>>>>>>en
>> >> >>>>>>d(
>> >> >>>>>>St
>> >> >>>>>>reamTask.java:517)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInput
>>>>>>>>>>Ou
>> >>>>>>>>tp
>> >> >>>>>>ut
>> >> >>>>>>(S
>> >> >>>>>>treamTask.java:171)
>> >> >>>>>> at
>> >> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>> >> >>>>>> ... 1 more
>> >> >>>>>> Caused by:
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$V
>>>>>>>>>>er
>> >>>>>>>>si
>> >> >>>>>>on
>> >> >>>>>>Mi
>> >> >>>>>>smatch):
>> >> >>>>>> Server IPC version 9 cannot communicate with client version 4
>> >> >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113)
>> >> >>>>>> at
>>org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> >> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
>> >>Source)
>> >> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >> >>>>>>Method)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorI
>>>>>>>>>>mp
>> >>>>>>>>l.
>> >> >>>>>>ja
>> >> >>>>>>va
>> >> >>>>>>:62)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
>>>>>>>>>>cc
>> >>>>>>>>es
>> >> >>>>>>so
>> >> >>>>>>rI
>> >> >>>>>>mpl.java:43)
>> >> >>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(Re
>>>>>>>>>>tr
>> >>>>>>>>yI
>> >> >>>>>>nv
>> >> >>>>>>oc
>> >> >>>>>>ationHandler.java:85)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInv
>>>>>>>>>>oc
>> >>>>>>>>at
>> >> >>>>>>io
>> >> >>>>>>nH
>> >> >>>>>>andler.java:62)
>> >> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
>> >>Source)
>> >> >>>>>> at
>>org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
>> >> >>>>>> at
>> >> >>>>>>
>> >>org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
>> >> >>>>>> at
>> >> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
>> >> >>>>>> at
>> >> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distribut
>>>>>>>>>>ed
>> >>>>>>>>Fi
>> >> >>>>>>le
>> >> >>>>>>Sy
>> >> >>>>>>stem.java:100)
>> >> >>>>>> at
>> >> >>>>>>
>> >>
>>
>>>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hado
>>>>>>>>>>op
>> >>>>>>>>Fi
>> >> >>>>>>le
>> >> >>>>>>Sy
>> >> >>>>>>stem.java:321)
>> >> >>>>>> ... 8 more
>> >> >>>>>>
>> >> >>>>>> I searched for this error online and it indicates that the
>>client
>> >> >>>>>>which
>> >> >>>>>>is
>> >> >>>>>> Flink in this case is at a much lower version. Is there a way
>>to
>> >> >>>>>>check
>> >> >>>>>>the
>> >> >>>>>> version of Hadoop packaged with my Flink installation?
>> >> >>>>>>
>> >> >>>>>> Thanks,
>> >> >>>>>> Ali
>> >> >>>>>>
>> >> >>>>
>> >> >>
>> >>
>> >>
>>
>>
Re: Error due to Hadoop version mismatch
Posted by Robert Metzger <rm...@apache.org>.
Okay, then, my previous statement is false. From the stack trace, it seems
that Flink is using an older Hadoop version. The DFSClient and RPC classes
look different in Hadoop 2.6.0.
Max checked already for some environment variables. Is $CLASSPATH set? Did
you install Hadoop only by downloading the binaries or did you use
something like the ubuntu package manager? (those packages sometimes place
jar files in some lib/ folders)
Did you put something additionally into the lib/ folder of Flink?
I think when Flink is starting up, its also logging the classpath of the
JVM. Can you post it here? (Maybe you need to start with DEBUG log level)
On Mon, Jan 4, 2016 at 8:46 PM, Kashmar, Ali <Al...@emc.com> wrote:
> Hi Robert,
>
> On the ubuntu host, the port 9000 is up:
>
> ubuntu@ubuntu-171:~$ netstat -tulpn
> ....
> tcp 0 0 10.13.182.171:9000 0.0.0.0:* LISTEN
> 8849/java
> ....
>
> And the process using this port is:
>
> ubuntu@ubuntu-171:~$ ps -ef | grep 8849
> ubuntu 8849 1 0 2015 ? 00:25:00
> /usr/lib/jvm/java-8-oracle/bin/java -Dproc_namenode -Xmx1000m
> -Djava.net.preferIPv4Stack=true
> -Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
> -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
> -Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
> -Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
> -Dhadoop.log.file=hadoop-ubuntu-namenode-ubuntu-171.log
> -Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
> -Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,RFA
> -Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
> -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
> -Dhadoop.security.logger=INFO,RFAS
> org.apache.hadoop.hdfs.server.namenode.NameNode
>
> And the hadoop version is:
>
> ubuntu@ubuntu-171:~$ Documents/hadoop-2.6.0/bin/hadoop version
> Hadoop 2.6.0
> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
> Compiled by jenkins on 2014-11-13T21:10Z
> Compiled with protoc 2.5.0
> From source with checksum 18e43357c8f927c0695f1e9522859d6a
> This command was run using
> /home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0
> .jar
>
>
>
> -Ali
>
>
>
> On 2016-01-04, 2:20 PM, "Robert Metzger" <rm...@apache.org> wrote:
>
> >Your Flink installation has Hadoop 2.6.0 included, on the other machine,
> >there is a Hadoop version installed, which is most likely a 1.x or even a
> >0.x version.
> >Are you sure that the host "ubuntu-171" has the ip 10.13.182.171 and that
> >the hadoop installation in the "/home/ubuntu/Documents/hadoop-2.6.0/"
> >directory is listening on port 9000 ?
> >
> >On Mon, Jan 4, 2016 at 3:03 PM, Kashmar, Ali <Al...@emc.com> wrote:
> >
> >> Hi Max,
> >>
> >> Both commands return nothing. Those variables aren’t set.
> >>
> >> The only software I installed on these machines is Flink and Java.
> >>
> >> -Ali
> >>
> >> On 2015-12-28, 6:42 AM, "Maximilian Michels" <mx...@apache.org> wrote:
> >>
> >> >Hi Ali,
> >> >
> >> >The warning about the native Hadoop libraries is nothing to worry
> >> >about. The native modules are platform-optimized modules which may be
> >> >used to improve performance. They are not necessary for Hadoop to
> >> >function correctly.
> >> >
> >> >The exception message implies that you are using is a very old version
> >> >of Hadoop. Do you have other Hadoop versions installed on the same
> >> >machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0
> >> >without any problems.
> >> >
> >> >On the cluster machines, what is the output of these commands?
> >> >
> >> >echo $HADOOP_CLASSPATH
> >> >echo $HADOOP_CONF_DIR
> >> >
> >> >
> >> >Thanks,
> >> >Max
> >> >
> >> >On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <Al...@emc.com>
> >> wrote:
> >> >> Hi Max,
> >> >>
> >> >> I have the same output for the Task Manager:
> >> >>
> >> >> 11:25:04,274 INFO org.apache.flink.runtime.taskmanager.TaskManager
> >> >> - Hadoop version: 2.6.0
> >> >>
> >> >> I do get this line at the beginning of both job and task manager log
> >> >>files:
> >> >>
> >> >> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader
> >> >> - Unable to load native-hadoop library for your platform...
> >>using
> >> >> builtin-java classes where applicable
> >> >>
> >> >> Do you think it has anything to do with it?
> >> >>
> >> >> Thanks,
> >> >> Ali
> >> >>
> >> >> On 2015-12-23, 7:30 AM, "Maximilian Michels" <mx...@apache.org> wrote:
> >> >>
> >> >>>Hi Ali,
> >> >>>
> >> >>>Could you please also post the Hadoop version output of the task
> >> >>>manager log files? It looks like the task managers are running a
> >> >>>different Hadoop version.
> >> >>>
> >> >>>Thanks,
> >> >>>Max
> >> >>>
> >> >>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali <Al...@emc.com>
> >> >>>wrote:
> >> >>>> Hi Robert,
> >> >>>>
> >> >>>> I found the version in the job manager log file:
> >> >>>>
> >> >>>> 17:33:49,636 INFO org.apache.flink.runtime.jobmanager.JobManager
> >> >>>> - Hadoop version: 2.6.0
> >> >>>>
> >> >>>> But the Hadoop installation I have is saying this:
> >> >>>>
> >> >>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version
> >> >>>> Hadoop 2.6.0
> >> >>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
> >> >>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
> >> >>>> Compiled by jenkins on 2014-11-13T21:10Z
> >> >>>> Compiled with protoc 2.5.0
> >> >>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a
> >> >>>> This command was run using
> >> >>>>
> >>
> >>>>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common
> >>>>>>-2
> >> >>>>.6
> >> >>>>.0
> >> >>>> .jar
> >> >>>>
> >> >>>>
> >> >>>> So one of them is lying to me? :)
> >> >>>>
> >> >>>> Ali
> >> >>>>
> >> >>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rm...@apache.org>
> >> wrote:
> >> >>>>
> >> >>>>>Hi Ali,
> >> >>>>>
> >> >>>>>the TaskManagers and the JobManager is logging the Hadoop version
> >>on
> >> >>>>>startup.
> >> >>>>>
> >> >>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali <Ali.Kashmar@emc.com
> >
> >> >>>>>wrote:
> >> >>>>>
> >> >>>>>> Hello,
> >> >>>>>>
> >> >>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I
> >> >>>>>>downloaded
> >> >>>>>>the
> >> >>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I also
> >> >>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and
> >>set
> >> >>>>>>up
> >> >>>>>>HDFS
> >> >>>>>> on a separate machine. When I start Flink I get the following
> >>error:
> >> >>>>>>
> >> >>>>>> 17:34:13,047 INFO org.apache.flink.runtime.jobmanager.JobManager
> >> >>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72 (Protocol
> >> >>>>>>Event
> >> >>>>>> Processing) changed to FAILED.
> >> >>>>>> java.lang.Exception: Call to registerInputOutput() of invokable
> >> >>>>>>failed
> >> >>>>>> at
> >> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
> >> >>>>>> at java.lang.Thread.run(Thread.java:745)
> >> >>>>>> Caused by: java.io.IOException: The given file URI (hdfs://
> >> >>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS
> >> >>>>>>NameNode
> >> >>>>>>at
> >> >>>>>> 10.13.182.171:9000, but the File System could not be initialized
> >> >>>>>>with
> >> >>>>>> that address: Server IPC version 9 cannot communicate with client
> >> >>>>>>version 4
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hadoop
> >>>>>>>>Fi
> >> >>>>>>le
> >> >>>>>>Sy
> >> >>>>>>stem.java:337)
> >> >>>>>> at
> >> >>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsS
> >>>>>>>>ta
> >> >>>>>>te
> >> >>>>>>Ba
> >> >>>>>>ckend.java:142)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsS
> >>>>>>>>ta
> >> >>>>>>te
> >> >>>>>>Ba
> >> >>>>>>ckend.java:101)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.cre
> >>>>>>>>at
> >> >>>>>>eF
> >> >>>>>>ro
> >> >>>>>>mConfig(FsStateBackendFactory.java:48)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBack
> >>>>>>>>en
> >> >>>>>>d(
> >> >>>>>>St
> >> >>>>>>reamTask.java:517)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOu
> >>>>>>>>tp
> >> >>>>>>ut
> >> >>>>>>(S
> >> >>>>>>treamTask.java:171)
> >> >>>>>> at
> >> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
> >> >>>>>> ... 1 more
> >> >>>>>> Caused by:
> >> >>>>>>
> >>
> >>>>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$Ver
> >>>>>>>>si
> >> >>>>>>on
> >> >>>>>>Mi
> >> >>>>>>smatch):
> >> >>>>>> Server IPC version 9 cannot communicate with client version 4
> >> >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113)
> >> >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> >> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
> >>Source)
> >> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>>>>Method)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp
> >>>>>>>>l.
> >> >>>>>>ja
> >> >>>>>>va
> >> >>>>>>:62)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
> >>>>>>>>es
> >> >>>>>>so
> >> >>>>>>rI
> >> >>>>>>mpl.java:43)
> >> >>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(Retr
> >>>>>>>>yI
> >> >>>>>>nv
> >> >>>>>>oc
> >> >>>>>>ationHandler.java:85)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvoc
> >>>>>>>>at
> >> >>>>>>io
> >> >>>>>>nH
> >> >>>>>>andler.java:62)
> >> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
> >>Source)
> >> >>>>>> at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
> >> >>>>>> at
> >> >>>>>>
> >>org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
> >> >>>>>> at
> >> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
> >> >>>>>> at
> >> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distributed
> >>>>>>>>Fi
> >> >>>>>>le
> >> >>>>>>Sy
> >> >>>>>>stem.java:100)
> >> >>>>>> at
> >> >>>>>>
> >>
> >>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hadoop
> >>>>>>>>Fi
> >> >>>>>>le
> >> >>>>>>Sy
> >> >>>>>>stem.java:321)
> >> >>>>>> ... 8 more
> >> >>>>>>
> >> >>>>>> I searched for this error online and it indicates that the client
> >> >>>>>>which
> >> >>>>>>is
> >> >>>>>> Flink in this case is at a much lower version. Is there a way to
> >> >>>>>>check
> >> >>>>>>the
> >> >>>>>> version of Hadoop packaged with my Flink installation?
> >> >>>>>>
> >> >>>>>> Thanks,
> >> >>>>>> Ali
> >> >>>>>>
> >> >>>>
> >> >>
> >>
> >>
>
>
Re: Error due to Hadoop version mismatch
Posted by "Kashmar, Ali" <Al...@emc.com>.
Hi Robert,
On the ubuntu host, the port 9000 is up:
ubuntu@ubuntu-171:~$ netstat -tulpn
....
tcp 0 0 10.13.182.171:9000 0.0.0.0:* LISTEN
8849/java
....
And the process using this port is:
ubuntu@ubuntu-171:~$ ps -ef | grep 8849
ubuntu 8849 1 0 2015 ? 00:25:00
/usr/lib/jvm/java-8-oracle/bin/java -Dproc_namenode -Xmx1000m
-Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
-Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,console
-Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true
-Dhadoop.log.dir=/home/ubuntu/Documents/hadoop-2.6.0/logs
-Dhadoop.log.file=hadoop-ubuntu-namenode-ubuntu-171.log
-Dhadoop.home.dir=/home/ubuntu/Documents/hadoop-2.6.0
-Dhadoop.id.str=ubuntu -Dhadoop.root.logger=INFO,RFA
-Djava.library.path=/home/ubuntu/Documents/hadoop-2.6.0/lib/native
-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender
-Dhadoop.security.logger=INFO,RFAS
org.apache.hadoop.hdfs.server.namenode.NameNode
And the hadoop version is:
ubuntu@ubuntu-171:~$ Documents/hadoop-2.6.0/bin/hadoop version
Hadoop 2.6.0
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
Compiled by jenkins on 2014-11-13T21:10Z
Compiled with protoc 2.5.0
From source with checksum 18e43357c8f927c0695f1e9522859d6a
This command was run using
/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2.6.0
.jar
-Ali
On 2016-01-04, 2:20 PM, "Robert Metzger" <rm...@apache.org> wrote:
>Your Flink installation has Hadoop 2.6.0 included, on the other machine,
>there is a Hadoop version installed, which is most likely a 1.x or even a
>0.x version.
>Are you sure that the host "ubuntu-171" has the ip 10.13.182.171 and that
>the hadoop installation in the "/home/ubuntu/Documents/hadoop-2.6.0/"
>directory is listening on port 9000 ?
>
>On Mon, Jan 4, 2016 at 3:03 PM, Kashmar, Ali <Al...@emc.com> wrote:
>
>> Hi Max,
>>
>> Both commands return nothing. Those variables aren’t set.
>>
>> The only software I installed on these machines is Flink and Java.
>>
>> -Ali
>>
>> On 2015-12-28, 6:42 AM, "Maximilian Michels" <mx...@apache.org> wrote:
>>
>> >Hi Ali,
>> >
>> >The warning about the native Hadoop libraries is nothing to worry
>> >about. The native modules are platform-optimized modules which may be
>> >used to improve performance. They are not necessary for Hadoop to
>> >function correctly.
>> >
>> >The exception message implies that you are using is a very old version
>> >of Hadoop. Do you have other Hadoop versions installed on the same
>> >machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0
>> >without any problems.
>> >
>> >On the cluster machines, what is the output of these commands?
>> >
>> >echo $HADOOP_CLASSPATH
>> >echo $HADOOP_CONF_DIR
>> >
>> >
>> >Thanks,
>> >Max
>> >
>> >On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <Al...@emc.com>
>> wrote:
>> >> Hi Max,
>> >>
>> >> I have the same output for the Task Manager:
>> >>
>> >> 11:25:04,274 INFO org.apache.flink.runtime.taskmanager.TaskManager
>> >> - Hadoop version: 2.6.0
>> >>
>> >> I do get this line at the beginning of both job and task manager log
>> >>files:
>> >>
>> >> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader
>> >> - Unable to load native-hadoop library for your platform...
>>using
>> >> builtin-java classes where applicable
>> >>
>> >> Do you think it has anything to do with it?
>> >>
>> >> Thanks,
>> >> Ali
>> >>
>> >> On 2015-12-23, 7:30 AM, "Maximilian Michels" <mx...@apache.org> wrote:
>> >>
>> >>>Hi Ali,
>> >>>
>> >>>Could you please also post the Hadoop version output of the task
>> >>>manager log files? It looks like the task managers are running a
>> >>>different Hadoop version.
>> >>>
>> >>>Thanks,
>> >>>Max
>> >>>
>> >>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali <Al...@emc.com>
>> >>>wrote:
>> >>>> Hi Robert,
>> >>>>
>> >>>> I found the version in the job manager log file:
>> >>>>
>> >>>> 17:33:49,636 INFO org.apache.flink.runtime.jobmanager.JobManager
>> >>>> - Hadoop version: 2.6.0
>> >>>>
>> >>>> But the Hadoop installation I have is saying this:
>> >>>>
>> >>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version
>> >>>> Hadoop 2.6.0
>> >>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
>> >>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
>> >>>> Compiled by jenkins on 2014-11-13T21:10Z
>> >>>> Compiled with protoc 2.5.0
>> >>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a
>> >>>> This command was run using
>> >>>>
>>
>>>>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common
>>>>>>-2
>> >>>>.6
>> >>>>.0
>> >>>> .jar
>> >>>>
>> >>>>
>> >>>> So one of them is lying to me? :)
>> >>>>
>> >>>> Ali
>> >>>>
>> >>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rm...@apache.org>
>> wrote:
>> >>>>
>> >>>>>Hi Ali,
>> >>>>>
>> >>>>>the TaskManagers and the JobManager is logging the Hadoop version
>>on
>> >>>>>startup.
>> >>>>>
>> >>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali <Al...@emc.com>
>> >>>>>wrote:
>> >>>>>
>> >>>>>> Hello,
>> >>>>>>
>> >>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I
>> >>>>>>downloaded
>> >>>>>>the
>> >>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I also
>> >>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and
>>set
>> >>>>>>up
>> >>>>>>HDFS
>> >>>>>> on a separate machine. When I start Flink I get the following
>>error:
>> >>>>>>
>> >>>>>> 17:34:13,047 INFO org.apache.flink.runtime.jobmanager.JobManager
>> >>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72 (Protocol
>> >>>>>>Event
>> >>>>>> Processing) changed to FAILED.
>> >>>>>> java.lang.Exception: Call to registerInputOutput() of invokable
>> >>>>>>failed
>> >>>>>> at
>> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
>> >>>>>> at java.lang.Thread.run(Thread.java:745)
>> >>>>>> Caused by: java.io.IOException: The given file URI (hdfs://
>> >>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS
>> >>>>>>NameNode
>> >>>>>>at
>> >>>>>> 10.13.182.171:9000, but the File System could not be initialized
>> >>>>>>with
>> >>>>>> that address: Server IPC version 9 cannot communicate with client
>> >>>>>>version 4
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hadoop
>>>>>>>>Fi
>> >>>>>>le
>> >>>>>>Sy
>> >>>>>>stem.java:337)
>> >>>>>> at
>> >>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsS
>>>>>>>>ta
>> >>>>>>te
>> >>>>>>Ba
>> >>>>>>ckend.java:142)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsS
>>>>>>>>ta
>> >>>>>>te
>> >>>>>>Ba
>> >>>>>>ckend.java:101)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.cre
>>>>>>>>at
>> >>>>>>eF
>> >>>>>>ro
>> >>>>>>mConfig(FsStateBackendFactory.java:48)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBack
>>>>>>>>en
>> >>>>>>d(
>> >>>>>>St
>> >>>>>>reamTask.java:517)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOu
>>>>>>>>tp
>> >>>>>>ut
>> >>>>>>(S
>> >>>>>>treamTask.java:171)
>> >>>>>> at
>> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
>> >>>>>> ... 1 more
>> >>>>>> Caused by:
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$Ver
>>>>>>>>si
>> >>>>>>on
>> >>>>>>Mi
>> >>>>>>smatch):
>> >>>>>> Server IPC version 9 cannot communicate with client version 4
>> >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113)
>> >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
>>Source)
>> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>>>>Method)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImp
>>>>>>>>l.
>> >>>>>>ja
>> >>>>>>va
>> >>>>>>:62)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
>>>>>>>>es
>> >>>>>>so
>> >>>>>>rI
>> >>>>>>mpl.java:43)
>> >>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(Retr
>>>>>>>>yI
>> >>>>>>nv
>> >>>>>>oc
>> >>>>>>ationHandler.java:85)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvoc
>>>>>>>>at
>> >>>>>>io
>> >>>>>>nH
>> >>>>>>andler.java:62)
>> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown
>>Source)
>> >>>>>> at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
>> >>>>>> at
>> >>>>>>
>>org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
>> >>>>>> at
>> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
>> >>>>>> at
>> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distributed
>>>>>>>>Fi
>> >>>>>>le
>> >>>>>>Sy
>> >>>>>>stem.java:100)
>> >>>>>> at
>> >>>>>>
>>
>>>>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(Hadoop
>>>>>>>>Fi
>> >>>>>>le
>> >>>>>>Sy
>> >>>>>>stem.java:321)
>> >>>>>> ... 8 more
>> >>>>>>
>> >>>>>> I searched for this error online and it indicates that the client
>> >>>>>>which
>> >>>>>>is
>> >>>>>> Flink in this case is at a much lower version. Is there a way to
>> >>>>>>check
>> >>>>>>the
>> >>>>>> version of Hadoop packaged with my Flink installation?
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Ali
>> >>>>>>
>> >>>>
>> >>
>>
>>
Re: Error due to Hadoop version mismatch
Posted by Robert Metzger <rm...@apache.org>.
Your Flink installation has Hadoop 2.6.0 included, on the other machine,
there is a Hadoop version installed, which is most likely a 1.x or even a
0.x version.
Are you sure that the host "ubuntu-171" has the ip 10.13.182.171 and that
the hadoop installation in the "/home/ubuntu/Documents/hadoop-2.6.0/"
directory is listening on port 9000 ?
On Mon, Jan 4, 2016 at 3:03 PM, Kashmar, Ali <Al...@emc.com> wrote:
> Hi Max,
>
> Both commands return nothing. Those variables aren’t set.
>
> The only software I installed on these machines is Flink and Java.
>
> -Ali
>
> On 2015-12-28, 6:42 AM, "Maximilian Michels" <mx...@apache.org> wrote:
>
> >Hi Ali,
> >
> >The warning about the native Hadoop libraries is nothing to worry
> >about. The native modules are platform-optimized modules which may be
> >used to improve performance. They are not necessary for Hadoop to
> >function correctly.
> >
> >The exception message implies that you are using is a very old version
> >of Hadoop. Do you have other Hadoop versions installed on the same
> >machine? We have had people using Flink 0.10.0 with Hadoop 2.6.0
> >without any problems.
> >
> >On the cluster machines, what is the output of these commands?
> >
> >echo $HADOOP_CLASSPATH
> >echo $HADOOP_CONF_DIR
> >
> >
> >Thanks,
> >Max
> >
> >On Wed, Dec 23, 2015 at 3:53 PM, Kashmar, Ali <Al...@emc.com>
> wrote:
> >> Hi Max,
> >>
> >> I have the same output for the Task Manager:
> >>
> >> 11:25:04,274 INFO org.apache.flink.runtime.taskmanager.TaskManager
> >> - Hadoop version: 2.6.0
> >>
> >> I do get this line at the beginning of both job and task manager log
> >>files:
> >>
> >> 11:25:04,100 WARN org.apache.hadoop.util.NativeCodeLoader
> >> - Unable to load native-hadoop library for your platform... using
> >> builtin-java classes where applicable
> >>
> >> Do you think it has anything to do with it?
> >>
> >> Thanks,
> >> Ali
> >>
> >> On 2015-12-23, 7:30 AM, "Maximilian Michels" <mx...@apache.org> wrote:
> >>
> >>>Hi Ali,
> >>>
> >>>Could you please also post the Hadoop version output of the task
> >>>manager log files? It looks like the task managers are running a
> >>>different Hadoop version.
> >>>
> >>>Thanks,
> >>>Max
> >>>
> >>>On Tue, Dec 22, 2015 at 4:28 PM, Kashmar, Ali <Al...@emc.com>
> >>>wrote:
> >>>> Hi Robert,
> >>>>
> >>>> I found the version in the job manager log file:
> >>>>
> >>>> 17:33:49,636 INFO org.apache.flink.runtime.jobmanager.JobManager
> >>>> - Hadoop version: 2.6.0
> >>>>
> >>>> But the Hadoop installation I have is saying this:
> >>>>
> >>>> ubuntu@ubuntu-171:~/Documents/hadoop-2.6.0$ bin/hadoop version
> >>>> Hadoop 2.6.0
> >>>> Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
> >>>> e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1
> >>>> Compiled by jenkins on 2014-11-13T21:10Z
> >>>> Compiled with protoc 2.5.0
> >>>> From source with checksum 18e43357c8f927c0695f1e9522859d6a
> >>>> This command was run using
> >>>>
> >>>>/home/ubuntu/Documents/hadoop-2.6.0/share/hadoop/common/hadoop-common-2
> >>>>.6
> >>>>.0
> >>>> .jar
> >>>>
> >>>>
> >>>> So one of them is lying to me? :)
> >>>>
> >>>> Ali
> >>>>
> >>>> On 2015-12-22, 10:16 AM, "Robert Metzger" <rm...@apache.org>
> wrote:
> >>>>
> >>>>>Hi Ali,
> >>>>>
> >>>>>the TaskManagers and the JobManager is logging the Hadoop version on
> >>>>>startup.
> >>>>>
> >>>>>On Tue, Dec 22, 2015 at 4:10 PM, Kashmar, Ali <Al...@emc.com>
> >>>>>wrote:
> >>>>>
> >>>>>> Hello,
> >>>>>>
> >>>>>> I¹m trying to use HDFS as store for Flink checkpoints so I
> >>>>>>downloaded
> >>>>>>the
> >>>>>> Hadoop 2.6.0/Scala 2.10 version of Flink and installed it. I also
> >>>>>> downloaded Hadoop 2.6.0 separately from the Hadoop website and set
> >>>>>>up
> >>>>>>HDFS
> >>>>>> on a separate machine. When I start Flink I get the following error:
> >>>>>>
> >>>>>> 17:34:13,047 INFO org.apache.flink.runtime.jobmanager.JobManager
> >>>>>> - Status of job 9ba32a08bc0ec02810bf5d2710842f72 (Protocol
> >>>>>>Event
> >>>>>> Processing) changed to FAILED.
> >>>>>> java.lang.Exception: Call to registerInputOutput() of invokable
> >>>>>>failed
> >>>>>> at
> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:529)
> >>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>> Caused by: java.io.IOException: The given file URI (hdfs://
> >>>>>> 10.13.182.171:9000/user/flink/checkpoints) points to the HDFS
> >>>>>>NameNode
> >>>>>>at
> >>>>>> 10.13.182.171:9000, but the File System could not be initialized
> >>>>>>with
> >>>>>> that address: Server IPC version 9 cannot communicate with client
> >>>>>>version 4
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFi
> >>>>>>le
> >>>>>>Sy
> >>>>>>stem.java:337)
> >>>>>> at
> >>>>>>org.apache.flink.core.fs.FileSystem.get(FileSystem.java:253)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsSta
> >>>>>>te
> >>>>>>Ba
> >>>>>>ckend.java:142)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsSta
> >>>>>>te
> >>>>>>Ba
> >>>>>>ckend.java:101)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.flink.runtime.state.filesystem.FsStateBackendFactory.creat
> >>>>>>eF
> >>>>>>ro
> >>>>>>mConfig(FsStateBackendFactory.java:48)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.createStateBacken
> >>>>>>d(
> >>>>>>St
> >>>>>>reamTask.java:517)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.flink.streaming.runtime.tasks.StreamTask.registerInputOutp
> >>>>>>ut
> >>>>>>(S
> >>>>>>treamTask.java:171)
> >>>>>> at
> >>>>>>org.apache.flink.runtime.taskmanager.Task.run(Task.java:526)
> >>>>>> ... 1 more
> >>>>>> Caused by:
> >>>>>>
> >>>>>>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.RPC$Versi
> >>>>>>on
> >>>>>>Mi
> >>>>>>smatch):
> >>>>>> Server IPC version 9 cannot communicate with client version 4
> >>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1113)
> >>>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source)
> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>Method)
> >>>>>> at
> >>>>>>
> >>>>>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>ja
> >>>>>>va
> >>>>>>:62)
> >>>>>> at
> >>>>>>
> >>>>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> >>>>>>so
> >>>>>>rI
> >>>>>>mpl.java:43)
> >>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
> >>>>>>nv
> >>>>>>oc
> >>>>>>ationHandler.java:85)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
> >>>>>>io
> >>>>>>nH
> >>>>>>andler.java:62)
> >>>>>> at com.sun.proxy.$Proxy6.getProtocolVersion(Unknown Source)
> >>>>>> at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
> >>>>>> at
> >>>>>> org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
> >>>>>> at
> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
> >>>>>> at
> >>>>>>org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFi
> >>>>>>le
> >>>>>>Sy
> >>>>>>stem.java:100)
> >>>>>> at
> >>>>>>
> >>>>>>org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFi
> >>>>>>le
> >>>>>>Sy
> >>>>>>stem.java:321)
> >>>>>> ... 8 more
> >>>>>>
> >>>>>> I searched for this error online and it indicates that the client
> >>>>>>which
> >>>>>>is
> >>>>>> Flink in this case is at a much lower version. Is there a way to
> >>>>>>check
> >>>>>>the
> >>>>>> version of Hadoop packaged with my Flink installation?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ali
> >>>>>>
> >>>>
> >>
>
>