You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by varuntango <va...@tangotech.co.in> on 2020/11/04 09:07:26 UTC
PutHDFS Datanode communication issue
Hi,
Im trying to put data into hdfs filesystem, but i didnt work properly. I
have configured both 50010, 8020 ports telnet everything working from nifi
server but still i cant be able to resolve this issue, Please anyone help me
out from this issue, it creates file in hdfs location but it didnt write the
content. Here is complete stacktrace;
2020-11-04 08:58:29,915 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@1bc8715e // Another save pending =
false
2020-11-04 08:58:33,594 INFO [NiFi Web Server-338]
o.a.n.c.queue.AbstractFlowFileQueue Initiating drop of FlowFiles from
FlowFileQueue[id=8e56b08a-0175-1000-32c1-44e0a0a2eb7c] on behalf of
anonymous (request identifier=927aa8fa-0175-1000-0956-7f6ab2993387)
2020-11-04 08:58:33,596 INFO [Drop FlowFiles for Connection
8e56b08a-0175-1000-32c1-44e0a0a2eb7c] o.a.n.c.queue.SwappablePriorityQueue
Successfully dropped 1 FlowFiles (11851 bytes) from Connection with ID
8e56b08a-0175-1000-32c1-44e0a0a2eb7c on behalf of anonymous
2020-11-04 08:58:38,507 INFO [NiFi Web Server-340]
o.a.n.c.s.StandardProcessScheduler Starting
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0]
2020-11-04 08:58:38,528 INFO [Timer-Driven Process Thread-10]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] to run with 1 threads
2020-11-04 08:58:38,564 WARN [Thread-416]
org.apache.hadoop.hdfs.DataStreamer DataStreamer Exception
java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:104)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:621)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at
org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
2020-11-04 08:58:39,492 INFO [Flow Service Tasks Thread-1]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@1bc8715e // Another save pending =
false
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: PutHDFS Datanode communication issue
Posted by varuntango <va...@tangotech.co.in>.
Hi Edward, Thanks for the reply here logger level into "DEBUG" mode updated
logs has been mentioned here, please help me to resolve this issue;
2020-11-05 05:29:23,029 INFO [NiFi Web Server-23]
o.a.n.c.s.StandardProcessScheduler Starting
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0]
2020-11-05 05:29:24,265 INFO [Flow Service Tasks Thread-1]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@1aa2ec6 // Another save pending =
false
2020-11-05 05:29:25,042 DEBUG [Timer-Driven Process Thread-10]
o.apache.nifi.processors.hadoop.PutHDFS
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] resetHDFSResources UGI root
(auth:SIMPLE)
2020-11-05 05:29:25,043 INFO [Timer-Driven Process Thread-10]
o.apache.nifi.processors.hadoop.PutHDFS
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] Initialized a new HDFS File
System with working dir:
hdfs://ec2-54-226-174-250.compute-1.amazonaws.com:8020/user/root default
block size: 134217728 default replication: 3 config: Configuration:
core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml,
yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml,
/home/opc/conf/hdfs-site.xml, /home/opc/conf/core-site.xml
2020-11-05 05:29:25,044 INFO [Timer-Driven Process Thread-10]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] to run with 1 threads
2020-11-05 05:29:25,337 WARN [Thread-115]
org.apache.hadoop.hdfs.DataStreamer DataStreamer Exception
java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:104)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:621)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at
org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
2020-11-05 05:29:25,349 INFO [Timer-Driven Process Thread-2]
o.apache.nifi.processors.hadoop.PutHDFS
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] copied
StandardFlowFileRecord[uuid=96cd4dd5-e1ff-4ca5-a8d5-5218de7f36ae,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1604554158431-1, container=default,
section=1], offset=0,
length=11851],offset=0,name=96cd4dd5-e1ff-4ca5-a8d5-5218de7f36ae,size=11851]
to HDFS at /retail-features-nifi/zztest/96cd4dd5-e1ff-4ca5-a8d5-5218de7f36ae
in 139 milliseconds at a rate of 82.96 KB/sec
2020-11-05 05:29:34,739 INFO [NiFi Web Server-16]
o.a.n.c.s.StandardProcessScheduler Stopping
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0]
2020-11-05 05:29:34,740 INFO [NiFi Web Server-16]
o.a.n.controller.StandardProcessorNode Stopping processor: class
org.apache.nifi.processors.hadoop.PutHDFS
2020-11-05 05:29:34,740 INFO [Timer-Driven Process Thread-4]
o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling
PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] to run
2020-11-05 05:29:34,749 WARN
[org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
org.apache.hadoop.fs.FileSystem Cleaner thread interrupted, will stop
java.lang.InterruptedException: null
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3740)
at java.lang.Thread.run(Thread.java:748)
2020-11-05 05:29:35,234 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@1aa2ec6 // Another save pending =
false
2020-11-05 05:30:37,266 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: PutHDFS Datanode communication issue
Posted by Edward Armes <ed...@gmail.com>.
Hi,
Looking at the error I would guess that for some reason the PutHDFS
processor isn't able to resolve the data node in HDFS.
Do you have any additional infornation around HDFS in your Nifi app log or
any information in the HDFS logs?
Otherwise I would suggest lowering lowering the log level for the HDFS
processor which can be configured in conf/logback.xml. That may give you
some more information.
Hope that helps
Edward
On Wed, 4 Nov 2020, 09:07 varuntango, <va...@tangotech.co.in> wrote:
> Hi,
>
> Im trying to put data into hdfs filesystem, but i didnt work properly. I
> have configured both 50010, 8020 ports telnet everything working from nifi
> server but still i cant be able to resolve this issue, Please anyone help
> me
> out from this issue, it creates file in hdfs location but it didnt write
> the
> content. Here is complete stacktrace;
>
> 2020-11-04 08:58:29,915 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@1bc8715e // Another save
> pending =
> false
> 2020-11-04 08:58:33,594 INFO [NiFi Web Server-338]
> o.a.n.c.queue.AbstractFlowFileQueue Initiating drop of FlowFiles from
> FlowFileQueue[id=8e56b08a-0175-1000-32c1-44e0a0a2eb7c] on behalf of
> anonymous (request identifier=927aa8fa-0175-1000-0956-7f6ab2993387)
> 2020-11-04 08:58:33,596 INFO [Drop FlowFiles for Connection
> 8e56b08a-0175-1000-32c1-44e0a0a2eb7c] o.a.n.c.queue.SwappablePriorityQueue
> Successfully dropped 1 FlowFiles (11851 bytes) from Connection with ID
> 8e56b08a-0175-1000-32c1-44e0a0a2eb7c on behalf of anonymous
> 2020-11-04 08:58:38,507 INFO [NiFi Web Server-340]
> o.a.n.c.s.StandardProcessScheduler Starting
> PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0]
> 2020-11-04 08:58:38,528 INFO [Timer-Driven Process Thread-10]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> PutHDFS[id=8e0a7636-0175-1000-810b-e0cb6cb164e0] to run with 1 threads
> 2020-11-04 08:58:38,564 WARN [Thread-416]
> org.apache.hadoop.hdfs.DataStreamer DataStreamer Exception
> java.nio.channels.UnresolvedAddressException: null
> at sun.nio.ch.Net.checkAddress(Net.java:104)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:621)
> at
> org.apache.hadoop.net
> .SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at
>
> org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
> at
>
> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
> at
>
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
> 2020-11-04 08:58:39,492 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@1bc8715e // Another save
> pending =
> false
>
>
>
>
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
>