You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by indus well <in...@gmail.com> on 2015/09/24 03:07:45 UTC
GetHTTP Usage with Dynamic Filenames
Hello NiFi Experts:
The GetHTTP processor works fine with static filename when getting files
from a website. However, I have a use case where I need to download a file
daily and the filename is the date of today, ie: 09222015.zip. Since the
URL property of the GetHTTP does not support expression language, I cannot
do something like http://example.com/${now():format('MMddyyyy')}.zip. Is
there a way I can specify the filename dynamically? Or using other
processor to make this work. Please advise.
Thanks,
Indus
Re: GetHTTP Usage with Dynamic Filenames
Posted by Joe Witt <jo...@gmail.com>.
well if you're willing to overlook the two jira's we uncovered for
this case and still think we're cool then you're kind of an awesome
user yourself!
Thanks
Joe
On Wed, Sep 23, 2015 at 9:08 PM, indus well <in...@gmail.com> wrote:
> Thanks, Joe. That did it, exactly what I was looking for.
>
> You guys and NiFi rock!
>
> On Wed, Sep 23, 2015 at 10:43 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Indus,
>>
>> We should probably have another JIRA to have the InvokeHTTP processor
>> capture response headers and store them as flow file attributes or at
>> least as a single attribute with the headers on it - something.
>> However, the core need you have is totally doable. I'd recommend the
>> following flow:
>>
>> GenerateFlowFile -> UpdateAttribute -> InvokeHTTP -> go do great
>> things with the content
>>
>> In UpdateAttribute create a property called 'filename' and give it a value
>> of
>> ${now():format('MMddyyyy')}.zip
>>
>> Then in the invoke HTTP use that for the expression language in the URL:
>> http://example.com/${filename}
>>
>> Now you'll get your content and have retained the filename you used to
>> pull it.
>>
>> Thanks
>> Joe
>>
>> On Wed, Sep 23, 2015 at 8:29 PM, indus well <in...@gmail.com> wrote:
>> > Worked like a charm! Thank you for your quick response, Daryl and Mark.
>> >
>> > The content of the downloaded file is stored in the file generated
>> > FlowFile
>> > with a random filename, how would I rename the output file to actual
>> > downloaded filename?
>> >
>> > By the way, the ticket is a good enhancement.
>> >
>> > Thanks,
>> >
>> > Indus
>> >
>> > On Wed, Sep 23, 2015 at 8:27 PM, Mark Payne <ma...@hotmail.com>
>> > wrote:
>> >>
>> >> Indus,
>> >>
>> >> As Daryl mentioned, InvokeHTTP will allow you to use the Expression
>> >> Language to do an HTTP GET.
>> >> It works a bit differently, though, because GetHTTP is a "Source
>> >> Processor" whereas InvokeHTTP needs to
>> >> be fed a FlowFile to do anything. So you can use GenerateFlowFile as a
>> >> source and have it generate a
>> >> 0 byte FlowFile (set the File Size property to "0 B"). Then just
>> >> connect
>> >> GenerateFlowFile to InvokeHTTP.
>> >>
>> >> Having said that, it is a bit awkward to have to use a GenerateFlowFile
>> >> to
>> >> trigger InvokeHTTP to run, so
>> >> I went ahead and created a ticket [1] to allow GetHTTP to evaluate the
>> >> Expression Language for the URL
>> >> property. In the meantime, though, GenerateFlowFile -> InvokeHTTP
>> >> should
>> >> provide you with
>> >> the capability you're looking for.
>> >>
>> >> Thanks
>> >> -Mark
>> >>
>> >> [1] https://issues.apache.org/jira/browse/NIFI-993
>> >>
>> >> On Sep 23, 2015, at 9:14 PM, Daryl Teo <da...@listcorp.com> wrote:
>> >>
>> >> Indus - use InvokeHTTP
>> >>
>> >> Daryl
>> >>
>> >> On Thu, Sep 24, 2015 at 11:07 AM, indus well <in...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hello NiFi Experts:
>> >>>
>> >>> The GetHTTP processor works fine with static filename when getting
>> >>> files
>> >>> from a website. However, I have a use case where I need to download a
>> >>> file
>> >>> daily and the filename is the date of today, ie: 09222015.zip. Since
>> >>> the URL
>> >>> property of the GetHTTP does not support expression language, I cannot
>> >>> do
>> >>> something like http://example.com/${now():format('MMddyyyy')}.zip. Is
>> >>> there
>> >>> a way I can specify the filename dynamically? Or using other processor
>> >>> to
>> >>> make this work. Please advise.
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Indus
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Daryl Teo
>> >>
>> >> Tech Lead
>> >>
>> >>
>> >> <logo2.jpg>
>> >>
>> >> 122 Cremorne Street
>> >> Richmond VIC 3121
>> >> Australia
>> >>
>> >> T +61 (3)8669 1821
>> >> M +61 (0)432 413 752
>> >>
>> >> www.listcorp.com
>> >>
>> >> <stroke.png>
>> >>
>> >> This email (including any attachments) is confidential and may be
>> >> privileged. It may be read, copied and used only by the intended
>> >> recipient.
>> >> If you have received it in error, please contact the sender immediately
>> >> by
>> >> return email. Please then delete both emails and do not disclose their
>> >> contents to any person. We believe, but do not warrant, that this email
>> >> and
>> >> any attachments are virus free. You should take full responsibility for
>> >> virus checking. Listcorp. reserves the right to monitor all email
>> >> communications. If the content of this email is personal or unconnected
>> >> with
>> >> our business, we accept no liability or responsibility for it.
>> >>
>> >>
>> >
>
>
Re: GetHTTP Usage with Dynamic Filenames
Posted by indus well <in...@gmail.com>.
Thanks, Joe. That did it, exactly what I was looking for.
You guys and NiFi rock!
On Wed, Sep 23, 2015 at 10:43 PM, Joe Witt <jo...@gmail.com> wrote:
> Indus,
>
> We should probably have another JIRA to have the InvokeHTTP processor
> capture response headers and store them as flow file attributes or at
> least as a single attribute with the headers on it - something.
> However, the core need you have is totally doable. I'd recommend the
> following flow:
>
> GenerateFlowFile -> UpdateAttribute -> InvokeHTTP -> go do great
> things with the content
>
> In UpdateAttribute create a property called 'filename' and give it a value
> of
> ${now():format('MMddyyyy')}.zip
>
> Then in the invoke HTTP use that for the expression language in the URL:
> http://example.com/${filename}
>
> Now you'll get your content and have retained the filename you used to
> pull it.
>
> Thanks
> Joe
>
> On Wed, Sep 23, 2015 at 8:29 PM, indus well <in...@gmail.com> wrote:
> > Worked like a charm! Thank you for your quick response, Daryl and Mark.
> >
> > The content of the downloaded file is stored in the file generated
> FlowFile
> > with a random filename, how would I rename the output file to actual
> > downloaded filename?
> >
> > By the way, the ticket is a good enhancement.
> >
> > Thanks,
> >
> > Indus
> >
> > On Wed, Sep 23, 2015 at 8:27 PM, Mark Payne <ma...@hotmail.com>
> wrote:
> >>
> >> Indus,
> >>
> >> As Daryl mentioned, InvokeHTTP will allow you to use the Expression
> >> Language to do an HTTP GET.
> >> It works a bit differently, though, because GetHTTP is a "Source
> >> Processor" whereas InvokeHTTP needs to
> >> be fed a FlowFile to do anything. So you can use GenerateFlowFile as a
> >> source and have it generate a
> >> 0 byte FlowFile (set the File Size property to "0 B"). Then just connect
> >> GenerateFlowFile to InvokeHTTP.
> >>
> >> Having said that, it is a bit awkward to have to use a GenerateFlowFile
> to
> >> trigger InvokeHTTP to run, so
> >> I went ahead and created a ticket [1] to allow GetHTTP to evaluate the
> >> Expression Language for the URL
> >> property. In the meantime, though, GenerateFlowFile -> InvokeHTTP should
> >> provide you with
> >> the capability you're looking for.
> >>
> >> Thanks
> >> -Mark
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-993
> >>
> >> On Sep 23, 2015, at 9:14 PM, Daryl Teo <da...@listcorp.com> wrote:
> >>
> >> Indus - use InvokeHTTP
> >>
> >> Daryl
> >>
> >> On Thu, Sep 24, 2015 at 11:07 AM, indus well <in...@gmail.com>
> wrote:
> >>>
> >>> Hello NiFi Experts:
> >>>
> >>> The GetHTTP processor works fine with static filename when getting
> files
> >>> from a website. However, I have a use case where I need to download a
> file
> >>> daily and the filename is the date of today, ie: 09222015.zip. Since
> the URL
> >>> property of the GetHTTP does not support expression language, I cannot
> do
> >>> something like http://example.com/${now():format('MMddyyyy')}.zip. Is
> there
> >>> a way I can specify the filename dynamically? Or using other processor
> to
> >>> make this work. Please advise.
> >>>
> >>> Thanks,
> >>>
> >>> Indus
> >>
> >>
> >>
> >>
> >> --
> >> Daryl Teo
> >>
> >> Tech Lead
> >>
> >>
> >> <logo2.jpg>
> >>
> >> 122 Cremorne Street
> >> Richmond VIC 3121
> >> Australia
> >>
> >> T +61 (3)8669 1821
> >> M +61 (0)432 413 752
> >>
> >> www.listcorp.com
> >>
> >> <stroke.png>
> >>
> >> This email (including any attachments) is confidential and may be
> >> privileged. It may be read, copied and used only by the intended
> recipient.
> >> If you have received it in error, please contact the sender immediately
> by
> >> return email. Please then delete both emails and do not disclose their
> >> contents to any person. We believe, but do not warrant, that this email
> and
> >> any attachments are virus free. You should take full responsibility for
> >> virus checking. Listcorp. reserves the right to monitor all email
> >> communications. If the content of this email is personal or unconnected
> with
> >> our business, we accept no liability or responsibility for it.
> >>
> >>
> >
>
Re: GetHTTP Usage with Dynamic Filenames
Posted by Joe Witt <jo...@gmail.com>.
Indus,
We should probably have another JIRA to have the InvokeHTTP processor
capture response headers and store them as flow file attributes or at
least as a single attribute with the headers on it - something.
However, the core need you have is totally doable. I'd recommend the
following flow:
GenerateFlowFile -> UpdateAttribute -> InvokeHTTP -> go do great
things with the content
In UpdateAttribute create a property called 'filename' and give it a value of
${now():format('MMddyyyy')}.zip
Then in the invoke HTTP use that for the expression language in the URL:
http://example.com/${filename}
Now you'll get your content and have retained the filename you used to pull it.
Thanks
Joe
On Wed, Sep 23, 2015 at 8:29 PM, indus well <in...@gmail.com> wrote:
> Worked like a charm! Thank you for your quick response, Daryl and Mark.
>
> The content of the downloaded file is stored in the file generated FlowFile
> with a random filename, how would I rename the output file to actual
> downloaded filename?
>
> By the way, the ticket is a good enhancement.
>
> Thanks,
>
> Indus
>
> On Wed, Sep 23, 2015 at 8:27 PM, Mark Payne <ma...@hotmail.com> wrote:
>>
>> Indus,
>>
>> As Daryl mentioned, InvokeHTTP will allow you to use the Expression
>> Language to do an HTTP GET.
>> It works a bit differently, though, because GetHTTP is a "Source
>> Processor" whereas InvokeHTTP needs to
>> be fed a FlowFile to do anything. So you can use GenerateFlowFile as a
>> source and have it generate a
>> 0 byte FlowFile (set the File Size property to "0 B"). Then just connect
>> GenerateFlowFile to InvokeHTTP.
>>
>> Having said that, it is a bit awkward to have to use a GenerateFlowFile to
>> trigger InvokeHTTP to run, so
>> I went ahead and created a ticket [1] to allow GetHTTP to evaluate the
>> Expression Language for the URL
>> property. In the meantime, though, GenerateFlowFile -> InvokeHTTP should
>> provide you with
>> the capability you're looking for.
>>
>> Thanks
>> -Mark
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-993
>>
>> On Sep 23, 2015, at 9:14 PM, Daryl Teo <da...@listcorp.com> wrote:
>>
>> Indus - use InvokeHTTP
>>
>> Daryl
>>
>> On Thu, Sep 24, 2015 at 11:07 AM, indus well <in...@gmail.com> wrote:
>>>
>>> Hello NiFi Experts:
>>>
>>> The GetHTTP processor works fine with static filename when getting files
>>> from a website. However, I have a use case where I need to download a file
>>> daily and the filename is the date of today, ie: 09222015.zip. Since the URL
>>> property of the GetHTTP does not support expression language, I cannot do
>>> something like http://example.com/${now():format('MMddyyyy')}.zip. Is there
>>> a way I can specify the filename dynamically? Or using other processor to
>>> make this work. Please advise.
>>>
>>> Thanks,
>>>
>>> Indus
>>
>>
>>
>>
>> --
>> Daryl Teo
>>
>> Tech Lead
>>
>>
>> <logo2.jpg>
>>
>> 122 Cremorne Street
>> Richmond VIC 3121
>> Australia
>>
>> T +61 (3)8669 1821
>> M +61 (0)432 413 752
>>
>> www.listcorp.com
>>
>> <stroke.png>
>>
>> This email (including any attachments) is confidential and may be
>> privileged. It may be read, copied and used only by the intended recipient.
>> If you have received it in error, please contact the sender immediately by
>> return email. Please then delete both emails and do not disclose their
>> contents to any person. We believe, but do not warrant, that this email and
>> any attachments are virus free. You should take full responsibility for
>> virus checking. Listcorp. reserves the right to monitor all email
>> communications. If the content of this email is personal or unconnected with
>> our business, we accept no liability or responsibility for it.
>>
>>
>
Re: GetHTTP Usage with Dynamic Filenames
Posted by indus well <in...@gmail.com>.
Worked like a charm! Thank you for your quick response, Daryl and Mark.
The content of the downloaded file is stored in the file generated FlowFile
with a random filename, how would I rename the output file to actual
downloaded filename?
By the way, the ticket is a good enhancement.
Thanks,
Indus
On Wed, Sep 23, 2015 at 8:27 PM, Mark Payne <ma...@hotmail.com> wrote:
> Indus,
>
> As Daryl mentioned, InvokeHTTP will allow you to use the Expression
> Language to do an HTTP GET.
> It works a bit differently, though, because GetHTTP is a "Source
> Processor" whereas InvokeHTTP needs to
> be fed a FlowFile to do anything. So you can use GenerateFlowFile as a
> source and have it generate a
> 0 byte FlowFile (set the File Size property to "0 B"). Then just connect
> GenerateFlowFile to InvokeHTTP.
>
> Having said that, it is a bit awkward to have to use a GenerateFlowFile to
> trigger InvokeHTTP to run, so
> I went ahead and created a ticket [1] to allow GetHTTP to evaluate the
> Expression Language for the URL
> property. In the meantime, though, GenerateFlowFile -> InvokeHTTP should
> provide you with
> the capability you're looking for.
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-993
>
> On Sep 23, 2015, at 9:14 PM, Daryl Teo <da...@listcorp.com> wrote:
>
> Indus - use InvokeHTTP
>
> Daryl
>
> On Thu, Sep 24, 2015 at 11:07 AM, indus well <in...@gmail.com> wrote:
>
>> Hello NiFi Experts:
>>
>> The GetHTTP processor works fine with static filename when getting files
>> from a website. However, I have a use case where I need to download a file
>> daily and the filename is the date of today, ie: 09222015.zip. Since the
>> URL property of the GetHTTP does not support expression language, I cannot
>> do something like http://example.com/${now():format('MMddyyyy')}.zip. Is
>> there a way I can specify the filename dynamically? Or using other
>> processor to make this work. Please advise.
>>
>> Thanks,
>>
>> Indus
>>
>
>
>
> --
> *Daryl Teo*
>
> Tech Lead
>
>
> <logo2.jpg>
>
> 122 Cremorne Street
> Richmond VIC 3121
> Australia
>
> *T *+61 (3)8669 1821
> *M * +61 (0)432 413 752
>
> www.listcorp.com
>
> <stroke.png>
>
> This email (including any attachments) is confidential and may be
> privileged. It may be read, copied and used only by the intended recipient.
> If you have received it in error, please contact the sender immediately by
> return email. Please then delete both emails and do not disclose their
> contents to any person. We believe, but do not warrant, that this email and
> any attachments are virus free. You should take full responsibility for
> virus checking. Listcorp. reserves the right to monitor all email
> communications. If the content of this email is personal or unconnected
> with our business, we accept no liability or responsibility for it.
>
>
>
Re: GetHTTP Usage with Dynamic Filenames
Posted by Mark Payne <ma...@hotmail.com>.
Indus,
As Daryl mentioned, InvokeHTTP will allow you to use the Expression Language to do an HTTP GET.
It works a bit differently, though, because GetHTTP is a "Source Processor" whereas InvokeHTTP needs to
be fed a FlowFile to do anything. So you can use GenerateFlowFile as a source and have it generate a
0 byte FlowFile (set the File Size property to "0 B"). Then just connect GenerateFlowFile to InvokeHTTP.
Having said that, it is a bit awkward to have to use a GenerateFlowFile to trigger InvokeHTTP to run, so
I went ahead and created a ticket [1] to allow GetHTTP to evaluate the Expression Language for the URL
property. In the meantime, though, GenerateFlowFile -> InvokeHTTP should provide you with
the capability you're looking for.
Thanks
-Mark
[1] https://issues.apache.org/jira/browse/NIFI-993 <https://issues.apache.org/jira/browse/NIFI-993>
> On Sep 23, 2015, at 9:14 PM, Daryl Teo <da...@listcorp.com> wrote:
>
> Indus - use InvokeHTTP
>
> Daryl
>
> On Thu, Sep 24, 2015 at 11:07 AM, indus well <induswell@gmail.com <ma...@gmail.com>> wrote:
> Hello NiFi Experts:
>
> The GetHTTP processor works fine with static filename when getting files from a website. However, I have a use case where I need to download a file daily and the filename is the date of today, ie: 09222015.zip. Since the URL property of the GetHTTP does not support expression language, I cannot do something like http://example.com/${now():format('MMddyyyy')}.zip <http://example.com/$%7Bnow():format('MMddyyyy')%7D.zip>. Is there a way I can specify the filename dynamically? Or using other processor to make this work. Please advise.
>
> Thanks,
>
> Indus
>
>
>
> --
> Daryl Teo
>
> Tech Lead
>
>
> <logo2.jpg>
>
> 122 Cremorne Street
> Richmond VIC 3121
> Australia
>
> T +61 (3)8669 1821
> M +61 (0)432 413 752
>
> www.listcorp.com <http://www.listcorp.com/>
> <stroke.png>
>
> This email (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error, please contact the sender immediately by return email. Please then delete both emails and do not disclose their contents to any person. We believe, but do not warrant, that this email and any attachments are virus free. You should take full responsibility for virus checking. Listcorp. reserves the right to monitor all email communications. If the content of this email is personal or unconnected with our business, we accept no liability or responsibility for it.
Re: GetHTTP Usage with Dynamic Filenames
Posted by Daryl Teo <da...@listcorp.com>.
Indus - use InvokeHTTP
Daryl
On Thu, Sep 24, 2015 at 11:07 AM, indus well <in...@gmail.com> wrote:
> Hello NiFi Experts:
>
> The GetHTTP processor works fine with static filename when getting files
> from a website. However, I have a use case where I need to download a file
> daily and the filename is the date of today, ie: 09222015.zip. Since the
> URL property of the GetHTTP does not support expression language, I cannot
> do something like http://example.com/${now():format('MMddyyyy')}.zip. Is
> there a way I can specify the filename dynamically? Or using other
> processor to make this work. Please advise.
>
> Thanks,
>
> Indus
>
--
*Daryl Teo*
Tech Lead
122 Cremorne Street
Richmond VIC 3121
Australia
*T *+61 (3)8669 1821
*M * +61 (0)432 413 752
www.listcorp.com
This email (including any attachments) is confidential and may be
privileged. It may be read, copied and used only by the intended recipient.
If you have received it in error, please contact the sender immediately by
return email. Please then delete both emails and do not disclose their
contents to any person. We believe, but do not warrant, that this email and
any attachments are virus free. You should take full responsibility for
virus checking. Listcorp. reserves the right to monitor all email
communications. If the content of this email is personal or unconnected
with our business, we accept no liability or responsibility for it.