You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by anup s <an...@philips.com> on 2015/05/18 11:44:48 UTC
losing property
Hi, I have the below workflow. I am intentionally failing the original zip
file, but when I put the uncompressed file into HDFS I'm losing on the
${path} property which is lost along with the original flow file. Is there
any way I can retain this property?
<http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/file/n1565/nifi-_%281%29.jpg>
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: losing property
Posted by anup s <an...@philips.com>.
Thanks Mark that worked.
I got another problem while writing it to HDFS. I don't think this is
related to that.
PutHDFS[id=104d8f8d-f5b3-35fa-81b2-1e761064a89b] Failed to write to HDFS due
to org.apache.nifi.processor.exception.ProcessException: IOException thrown
from PutHDFS[id=104d8f8d-f5b3-35fa-81b2-1e761064a89b]:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on /user/RS/nifi/ingest/1074-5-6/.log201210310000.zip (inode
1860488): File does not exist. [Lease. Holder:
DFSClient_NONMAPREDUCE_-856658733_73, pendingcreates: 1]
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2991)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2796)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2680)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:569)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
Any idea how do I get pass through this?
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1588.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: losing property
Posted by anup s <an...@philips.com>.
Thanks Mark! That worked..
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1589.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: losing property
Posted by Mark Payne <ma...@hotmail.com>.
OK cool. Just wanted to make sure that I don't give bad advice.
So when UnpackContent unpacks a zip file, it set a new value for the path attribute based on the path within the .zip file.
So in order to avoid this, what I would do is add an UpdateAttribute before UnpackContent. Configure UpdateAttribute with a new property. For the property name, use "zipfile.path" and for the value, use "${path}". So basically you're just copying the value of "path" to "zipfile.path". Then, in PutHDFS, for the directory, you can use "${zipfile.path}". Or, if you want to also include the path of the file within the .zip, you could use "${zipfile.path}/${path}"
Does that make sense?
Thanks
-Mark
----------------------------------------
> Date: Mon, 18 May 2015 07:54:18 -0700
> From: anup.sethuram@philips.com
> To: dev@nifi.incubator.apache.org
> Subject: RE: losing property
>
> That's right Mark! I want to retain the directory structure.
>
>
>
> --
> View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1577.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: losing property
Posted by anup s <an...@philips.com>.
That's right Mark! I want to retain the directory structure.
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1577.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: losing property
Posted by Mark Payne <ma...@hotmail.com>.
OK, I think I understand now. So you want to put the unzipped file to HDFS but use the path that was assigned to the zip file?
----------------------------------------
> Date: Mon, 18 May 2015 05:20:23 -0700
> From: anup.sethuram@philips.com
> To: dev@nifi.incubator.apache.org
> Subject: RE: losing property
>
> Hi Mark,
> I am auto-terminating the orginal zip flow file. And on success (viz.
> uncompression) the unzipped file goes to HDFS and puts the file at the new
> location. But the directory structure isn't created, althougth the Put HDFS
> file location is mentioned as <HDFS loc>/${path}.
>
> In the case, where the source is not a zip file the directory is created and
> moved under the directory, but this isn't working for the unpack route..
>
>
>
> --
> View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1570.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: losing property
Posted by anup s <an...@philips.com>.
Hi Mark,
I am auto-terminating the orginal zip flow file. And on success (viz.
uncompression) the unzipped file goes to HDFS and puts the file at the new
location. But the directory structure isn't created, althougth the Put HDFS
file location is mentioned as <HDFS loc>/${path}.
In the case, where the source is not a zip file the directory is created and
moved under the directory, but this isn't working for the unpack route..
--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565p1570.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
RE: losing property
Posted by Mark Payne <ma...@hotmail.com>.
Anup,
I'm not sure that I understand what you mean when you say you're "losing on the ${path} property which is lost along with the original flow file". Can you explain a bit further? The FlowFIle should be transferred to the 'success' relationship after sending to HDFS, and all attributes should be kept intact.
Thanks
-Mark
----------------------------------------
> Date: Mon, 18 May 2015 02:44:48 -0700
> From: anup.sethuram@philips.com
> To: dev@nifi.incubator.apache.org
> Subject: losing property
>
> Hi, I have the below workflow. I am intentionally failing the original zip
> file, but when I put the uncompressed file into HDFS I'm losing on the
> ${path} property which is lost along with the original flow file. Is there
> any way I can retain this property?
> <http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/file/n1565/nifi-_%281%29.jpg>
>
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/losing-property-tp1565.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.