You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Ben Meng <Be...@lifelock.com> on 2015/10/15 21:38:06 UTC
Provenance doesn't work with FetchS3Object
I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object.
"Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)"
Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile?
Regards,
Ben
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: Provenance doesn't work with FetchS3Object
Posted by Ben Meng <Be...@lifelock.com>.
Thanks Mark. That makes sense.
Regards,
Ben
From: Mark Payne
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Date: Thursday, October 15, 2015 at 1:09 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Subject: Re: Provenance doesn't work with FetchS3Object
Ben,
Since FetchS3Object is not creating the FlowFile, it should not be emitting a RECEIVE event. This is certainly a bug.
I have created a ticket for this: https://issues.apache.org/jira/browse/NIFI-1038
Thanks
-Mark
On Oct 15, 2015, at 3:57 PM, Ben Meng <Be...@lifelock.com>> wrote:
Oleg,
Yes, I’ve tried running FetchS3Object without any incoming FlowFile, and it just didn’t generate any output. I’ve also confirmed the behavior by inspecting the codes. The first thing it does is to check if there’s an existing FlowFile, and returns if there isn’t.
Regards,
Ben
From: Oleg Zhurakousky
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Date: Thursday, October 15, 2015 at 12:49 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Subject: Re: Provenance doesn't work with FetchS3Object
Ben
I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling.
Have you tried it without incoming FlowFiles?
Cheers
Oleg
On Oct 15, 2015, at 3:38 PM, Ben Meng <Be...@lifelock.com>> wrote:
I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object.
"Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)"
Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile?
Regards,
Ben
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: Provenance doesn't work with FetchS3Object
Posted by Mark Payne <ma...@hotmail.com>.
Ben,
Since FetchS3Object is not creating the FlowFile, it should not be emitting a RECEIVE event. This is certainly a bug.
I have created a ticket for this: https://issues.apache.org/jira/browse/NIFI-1038 <https://issues.apache.org/jira/browse/NIFI-1038>
Thanks
-Mark
> On Oct 15, 2015, at 3:57 PM, Ben Meng <Be...@lifelock.com> wrote:
>
> Oleg,
>
> Yes, I’ve tried running FetchS3Object without any incoming FlowFile, and it just didn’t generate any output. I’ve also confirmed the behavior by inspecting the codes. The first thing it does is to check if there’s an existing FlowFile, and returns if there isn’t.
>
> Regards,
> Ben
>
>
> From: Oleg Zhurakousky
> Reply-To: "users@nifi.apache.org <ma...@nifi.apache.org>"
> Date: Thursday, October 15, 2015 at 12:49 PM
> To: "users@nifi.apache.org <ma...@nifi.apache.org>"
> Subject: Re: Provenance doesn't work with FetchS3Object
>
> Ben
>
> I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling.
> Have you tried it without incoming FlowFiles?
>
> Cheers
> Oleg
>
>> On Oct 15, 2015, at 3:38 PM, Ben Meng <Ben.Meng@lifelock.com <ma...@lifelock.com>> wrote:
>>
>> I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object.
>>
>> "Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)"
>>
>> Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile?
>>
>> Regards,
>> Ben
>> The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
>
> The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Re: Provenance doesn't work with FetchS3Object
Posted by Ben Meng <Be...@lifelock.com>.
Oleg,
Yes, I’ve tried running FetchS3Object without any incoming FlowFile, and it just didn’t generate any output. I’ve also confirmed the behavior by inspecting the codes. The first thing it does is to check if there’s an existing FlowFile, and returns if there isn’t.
Regards,
Ben
From: Oleg Zhurakousky
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Date: Thursday, October 15, 2015 at 12:49 PM
To: "users@nifi.apache.org<ma...@nifi.apache.org>"
Subject: Re: Provenance doesn't work with FetchS3Object
Ben
I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling.
Have you tried it without incoming FlowFiles?
Cheers
Oleg
On Oct 15, 2015, at 3:38 PM, Ben Meng <Be...@lifelock.com>> wrote:
I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object.
"Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)"
Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile?
Regards,
Ben
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: Provenance doesn't work with FetchS3Object
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Ben
I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling.
Have you tried it without incoming FlowFiles?
Cheers
Oleg
On Oct 15, 2015, at 3:38 PM, Ben Meng <Be...@lifelock.com>> wrote:
I understand that FetchS3Object processor requires an incoming FlowFile to trigger it. The problem is that FetchS3Object emits a RECEIVE provenance event for the existing FlowFile. That event causes following error when I try to open the lineage chart for a simple flow: GenerateFlowFile -> FetchS3Object.
"Found cycle in graph. This indicates that multiple events were registered claiming to have generated the same FlowFile (UUID = 40f58407-ea10-4843-b8d1-be0e24f685aa)"
Should FetchS3Object create a new FlowFile for each fetched object? If so, does it really require an incoming FlowFile?
Regards,
Ben
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________