You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Joe Skora <js...@gmail.com> on 2015/10/08 12:38:39 UTC
Re: FetchS3Object processor
Chakri,
The Amazon docs explain here
<http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html> that
S3 doesn't support pathes within a bucket, but you can embed path-like
naming into the "Object Key" parameter to FetchS3Object.
For your example, you would use bucket="s3://<bucket>" and
key="<path>/<objectfile>" to incorporate the path as you store the object.
I hope that helps.
Regards,
Joe
On Thu, Oct 8, 2015 at 2:39 AM, Chakrader Dewaragatla <
Chakrader.Dewaragatla@lifelock.com> wrote:
> Nifi users - FetchS3Object processor do not have option to specify Object
> file path, how do you set that ?
>
> Eg : s3://<bucket>/<path>/<objectfile>
>
> Thanks,
> -Chakri
> ------------------------------
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ------------------------------
>
Re: FetchS3Object processor
Posted by Chakrader Dewaragatla <Ch...@lifelock.com>.
Thanks Joe, I was about to request a feature like this (ListS3Objects).
From: Joe Skora <js...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, October 8, 2015 at 11:39 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: FetchS3Object processor
There is already a ticket (NIFI-840<https://issues.apache.org/jira/browse/NIFI-840>) in the hopper to create a ListS3Objects processor that can track bucket contents and trigger FetchS3Object.
In the meantime, if you want it triggered automatically (as opposed to generate flow file) you may be able to accomplish something similar with a combination of HTTP and XML processors.
On Thu, Oct 8, 2015 at 2:32 PM, Chakrader Dewaragatla <Ch...@lifelock.com>> wrote:
Thanks Aldrin. GenerateFlowFile helped to trigger.
Thanks,
-Chakri
From: Aldrin Piri <al...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, October 8, 2015 at 11:22 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: FetchS3Object processor
Chakri,
The FetchS3Object processor does not act as a source directly and is instead driven by incoming FlowFiles. The use case was that one could utilize Expression Language to interact with buckets and paths in a dynamic fashion. Alternatively, if you are looking to just grab a specific file, you can use GenerateFlowFile to "trigger" the FetchS3Object with its static configuration.
On Thu, Oct 8, 2015 at 1:45 PM, Chakrader Dewaragatla <Ch...@lifelock.com>> wrote:
Thanks Joe.
I have S3fetch and putfile processors setup, files are not fetching despite files exist on s3. Am I missing anything ? Does S3fetch processor need any additional processor to feed data ?
Thanks,
-Chakri
From: Joe Skora <js...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, October 8, 2015 at 3:38 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: FetchS3Object processor
Chakri,
The Amazon docs explain here<http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html> that S3 doesn't support pathes within a bucket, but you can embed path-like naming into the "Object Key" parameter to FetchS3Object.
For your example, you would use bucket="s3://<bucket>" and key="<path>/<objectfile>" to incorporate the path as you store the object.
I hope that helps.
Regards,
Joe
On Thu, Oct 8, 2015 at 2:39 AM, Chakrader Dewaragatla <Ch...@lifelock.com>> wrote:
Nifi users - FetchS3Object processor do not have option to specify Object file path, how do you set that ?
Eg : s3://<bucket>/<path>/<objectfile>
Thanks,
-Chakri
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: FetchS3Object processor
Posted by Joe Skora <js...@gmail.com>.
There is already a ticket (NIFI-840
<https://issues.apache.org/jira/browse/NIFI-840>) in the hopper to create a
ListS3Objects processor that can track bucket contents and trigger
FetchS3Object.
In the meantime, if you want it triggered automatically (as opposed to
generate flow file) you may be able to accomplish something similar with a
combination of HTTP and XML processors.
On Thu, Oct 8, 2015 at 2:32 PM, Chakrader Dewaragatla <
Chakrader.Dewaragatla@lifelock.com> wrote:
> Thanks Aldrin. GenerateFlowFile helped to trigger.
>
> Thanks,
> -Chakri
>
> From: Aldrin Piri <al...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, October 8, 2015 at 11:22 AM
>
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: FetchS3Object processor
>
> Chakri,
>
> The FetchS3Object processor does not act as a source directly and is
> instead driven by incoming FlowFiles. The use case was that one could
> utilize Expression Language to interact with buckets and paths in a dynamic
> fashion. Alternatively, if you are looking to just grab a specific file,
> you can use GenerateFlowFile to "trigger" the FetchS3Object with its static
> configuration.
>
> On Thu, Oct 8, 2015 at 1:45 PM, Chakrader Dewaragatla <
> Chakrader.Dewaragatla@lifelock.com> wrote:
>
>> Thanks Joe.
>> I have S3fetch and putfile processors setup, files are not fetching
>> despite files exist on s3. Am I missing anything ? Does S3fetch processor
>> need any additional processor to feed data ?
>>
>> Thanks,
>> -Chakri
>>
>>
>> From: Joe Skora <js...@gmail.com>
>> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Date: Thursday, October 8, 2015 at 3:38 AM
>> To: "users@nifi.apache.org" <us...@nifi.apache.org>
>> Subject: Re: FetchS3Object processor
>>
>> Chakri,
>>
>> The Amazon docs explain here
>> <http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html>
>> that S3 doesn't support pathes within a bucket, but you can embed path-like
>> naming into the "Object Key" parameter to FetchS3Object.
>>
>> For your example, you would use bucket="s3://<bucket>" and
>> key="<path>/<objectfile>" to incorporate the path as you store the object.
>>
>> I hope that helps.
>>
>> Regards,
>> Joe
>>
>> On Thu, Oct 8, 2015 at 2:39 AM, Chakrader Dewaragatla <
>> Chakrader.Dewaragatla@lifelock.com> wrote:
>>
>>> Nifi users - FetchS3Object processor do not have option to specify
>>> Object file path, how do you set that ?
>>>
>>> Eg : s3://<bucket>/<path>/<objectfile>
>>>
>>> Thanks,
>>> -Chakri
>>> ------------------------------
>>> The information contained in this transmission may contain privileged
>>> and confidential information. It is intended only for the use of the
>>> person(s) named above. If you are not the intended recipient, you are
>>> hereby notified that any review, dissemination, distribution or duplication
>>> of this communication is strictly prohibited. If you are not the intended
>>> recipient, please contact the sender by reply email and destroy all copies
>>> of the original message.
>>> ------------------------------
>>>
>>
>> ------------------------------
>> The information contained in this transmission may contain privileged and
>> confidential information. It is intended only for the use of the person(s)
>> named above. If you are not the intended recipient, you are hereby notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all copies
>> of the original message.
>> ------------------------------
>>
>
> ------------------------------
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ------------------------------
>
Re: FetchS3Object processor
Posted by Chakrader Dewaragatla <Ch...@lifelock.com>.
Thanks Aldrin. GenerateFlowFile helped to trigger.
Thanks,
-Chakri
From: Aldrin Piri <al...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, October 8, 2015 at 11:22 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: FetchS3Object processor
Chakri,
The FetchS3Object processor does not act as a source directly and is instead driven by incoming FlowFiles. The use case was that one could utilize Expression Language to interact with buckets and paths in a dynamic fashion. Alternatively, if you are looking to just grab a specific file, you can use GenerateFlowFile to "trigger" the FetchS3Object with its static configuration.
On Thu, Oct 8, 2015 at 1:45 PM, Chakrader Dewaragatla <Ch...@lifelock.com>> wrote:
Thanks Joe.
I have S3fetch and putfile processors setup, files are not fetching despite files exist on s3. Am I missing anything ? Does S3fetch processor need any additional processor to feed data ?
Thanks,
-Chakri
From: Joe Skora <js...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, October 8, 2015 at 3:38 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: FetchS3Object processor
Chakri,
The Amazon docs explain here<http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html> that S3 doesn't support pathes within a bucket, but you can embed path-like naming into the "Object Key" parameter to FetchS3Object.
For your example, you would use bucket="s3://<bucket>" and key="<path>/<objectfile>" to incorporate the path as you store the object.
I hope that helps.
Regards,
Joe
On Thu, Oct 8, 2015 at 2:39 AM, Chakrader Dewaragatla <Ch...@lifelock.com>> wrote:
Nifi users - FetchS3Object processor do not have option to specify Object file path, how do you set that ?
Eg : s3://<bucket>/<path>/<objectfile>
Thanks,
-Chakri
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
Re: FetchS3Object processor
Posted by Aldrin Piri <al...@gmail.com>.
Chakri,
The FetchS3Object processor does not act as a source directly and is
instead driven by incoming FlowFiles. The use case was that one could
utilize Expression Language to interact with buckets and paths in a dynamic
fashion. Alternatively, if you are looking to just grab a specific file,
you can use GenerateFlowFile to "trigger" the FetchS3Object with its static
configuration.
On Thu, Oct 8, 2015 at 1:45 PM, Chakrader Dewaragatla <
Chakrader.Dewaragatla@lifelock.com> wrote:
> Thanks Joe.
> I have S3fetch and putfile processors setup, files are not fetching
> despite files exist on s3. Am I missing anything ? Does S3fetch processor
> need any additional processor to feed data ?
>
> Thanks,
> -Chakri
>
>
> From: Joe Skora <js...@gmail.com>
> Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Date: Thursday, October 8, 2015 at 3:38 AM
> To: "users@nifi.apache.org" <us...@nifi.apache.org>
> Subject: Re: FetchS3Object processor
>
> Chakri,
>
> The Amazon docs explain here
> <http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html>
> that S3 doesn't support pathes within a bucket, but you can embed path-like
> naming into the "Object Key" parameter to FetchS3Object.
>
> For your example, you would use bucket="s3://<bucket>" and
> key="<path>/<objectfile>" to incorporate the path as you store the object.
>
> I hope that helps.
>
> Regards,
> Joe
>
> On Thu, Oct 8, 2015 at 2:39 AM, Chakrader Dewaragatla <
> Chakrader.Dewaragatla@lifelock.com> wrote:
>
>> Nifi users - FetchS3Object processor do not have option to specify
>> Object file path, how do you set that ?
>>
>> Eg : s3://<bucket>/<path>/<objectfile>
>>
>> Thanks,
>> -Chakri
>> ------------------------------
>> The information contained in this transmission may contain privileged and
>> confidential information. It is intended only for the use of the person(s)
>> named above. If you are not the intended recipient, you are hereby notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all copies
>> of the original message.
>> ------------------------------
>>
>
> ------------------------------
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> ------------------------------
>
Re: FetchS3Object processor
Posted by Chakrader Dewaragatla <Ch...@lifelock.com>.
Thanks Joe.
I have S3fetch and putfile processors setup, files are not fetching despite files exist on s3. Am I missing anything ? Does S3fetch processor need any additional processor to feed data ?
Thanks,
-Chakri
From: Joe Skora <js...@gmail.com>>
Reply-To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Date: Thursday, October 8, 2015 at 3:38 AM
To: "users@nifi.apache.org<ma...@nifi.apache.org>" <us...@nifi.apache.org>>
Subject: Re: FetchS3Object processor
Chakri,
The Amazon docs explain here<http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html> that S3 doesn't support pathes within a bucket, but you can embed path-like naming into the "Object Key" parameter to FetchS3Object.
For your example, you would use bucket="s3://<bucket>" and key="<path>/<objectfile>" to incorporate the path as you store the object.
I hope that helps.
Regards,
Joe
On Thu, Oct 8, 2015 at 2:39 AM, Chakrader Dewaragatla <Ch...@lifelock.com>> wrote:
Nifi users - FetchS3Object processor do not have option to specify Object file path, how do you set that ?
Eg : s3://<bucket>/<path>/<objectfile>
Thanks,
-Chakri
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________
________________________________
The information contained in this transmission may contain privileged and confidential information. It is intended only for the use of the person(s) named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
________________________________