You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Dweep Sharma <dw...@redbus.com> on 2020/05/19 14:45:27 UTC
Combine Attributes & Content
Hi All,
Does anyone have a strategy to combine Attributes & Content into a single
Flowfile?
AttributesToJson seems like the best way convert all attributes to content
-Json but I
lose the original content.
Basically, I have a HandleHTTPRequest Processor and headers and body must
be combined,
currently Headers are stored as Attributes in a flowfile
-Dweep
--
*::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachments are confidential and
intended for the named recipient(s) only.E-mail transmission is not
guaranteed to be secure or error-free as information could be intercepted,
corrupted,lost, destroyed, arrive late or incomplete, or may contain
viruses in transmission. The e mail and its contents(with or without
referred errors) shall therefore not attach any liability on the originator
or redBus.com. Views or opinions, if any, presented in this email are
solely those of the author and may not necessarily reflect the views or
opinions of redBus.com. Any form of reproduction, dissemination, copying,
disclosure, modification,distribution and / or publication of this message
without the prior written consent of authorized representative of redbus.
<http://redbus.in/>com is strictly prohibited. If you have received this
email in error please delete it and notify the sender immediately.Before
opening any email and/or attachments, please check them for viruses and
other defects.*
Re: Combine Attributes & Content
Posted by Dweep Sharma <dw...@redbus.com>.
Thanks, I will try this approach as well. I can combine this with some
other flows/processes I have in mind.
On Thu, May 21, 2020 at 3:00 PM Edward Armes <ed...@gmail.com> wrote:
> Hi Sharma,
>
> An alternative could be, duplicate the FlowFile from the HandleHTTPRequest
> Processor have one wait while the other goes through an AttributesToJSON
> and stores the JSON in a lookup cache. The other FlowFile then grabs that
> json from the lookup cache and then use ReplaceText processor mode to add
> that specific attribute into the content.
>
> Edward
>
> On Thu, May 21, 2020 at 10:11 AM Dweep Sharma <dw...@redbus.com>
> wrote:
>
>> Thanks Matt, UpdateRecord works however, all headers must be added
>> manually (unlike AttributesToJson). Since it is one time effort, it will do
>> for now.
>>
>> -Dweep
>>
>> On Tue, May 19, 2020 at 9:16 PM Matt Burgess <ma...@apache.org>
>> wrote:
>>
>>> Dweep,
>>>
>>> Depending on how complex the content JSON is, you might be able to use
>>> ReplaceText to smuggle the attributes into the text, but this can be
>>> tricky as you need to match on the opening JSON and the rest, and then
>>> replace it with the opening JSON, the attributes, then the rest in
>>> order to preserve valid JSON.
>>>
>>> You may also be able to use UpdateRecord to add fields to your
>>> content, I believe if the Replacement Value Strategy property is set
>>> to "Literal Value" you can use Expression Language which may be able
>>> access flow file attributes.
>>>
>>> An alternative is to use a scripted processor if you're comfortable
>>> with a scripting language such as Groovy, as it can "slurp" JSON into
>>> a Plain Old Groovy Object (POGO), then you can add fields and
>>> serialize back out to JSON. I have a somewhat similar example here
>>> [1].
>>>
>>> Regards,
>>> Matt
>>>
>>> [1]
>>> http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
>>>
>>> On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <dw...@redbus.com>
>>> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > Does anyone have a strategy to combine Attributes & Content into a
>>> single Flowfile?
>>> >
>>> > AttributesToJson seems like the best way convert all attributes to
>>> content -Json but I
>>> >
>>> > lose the original content.
>>> >
>>> > Basically, I have a HandleHTTPRequest Processor and headers and body
>>> must be combined,
>>> >
>>> > currently Headers are stored as Attributes in a flowfile
>>> >
>>> > -Dweep
>>> >
>>> >
>>> > ::DISCLAIMER::
>>> >
>>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>> >
>>> > The contents of this e-mail and any attachments are confidential and
>>> intended for the named recipient(s) only.E-mail transmission is not
>>> guaranteed to be secure or error-free as information could be intercepted,
>>> corrupted,lost, destroyed, arrive late or incomplete, or may contain
>>> viruses in transmission. The e mail and its contents(with or without
>>> referred errors) shall therefore not attach any liability on the originator
>>> or redBus.com. Views or opinions, if any, presented in this email are
>>> solely those of the author and may not necessarily reflect the views or
>>> opinions of redBus.com. Any form of reproduction, dissemination, copying,
>>> disclosure, modification,distribution and / or publication of this message
>>> without the prior written consent of authorized representative of
>>> redbus.com is strictly prohibited. If you have received this email in
>>> error please delete it and notify the sender immediately.Before opening any
>>> email and/or attachments, please check them for viruses and other defects.
>>>
>>
>>
>>
>>
>> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The
>> contents of this e-mail and any attachments are confidential and intended
>> for the named recipient(s) only.E-mail transmission is not guaranteed to be
>> secure or error-free as information could be intercepted, corrupted,lost,
>> destroyed, arrive late or incomplete, or may contain viruses in
>> transmission. The e mail and its contents(with or without referred errors)
>> shall therefore not attach any liability on the originator or redBus.com.
>> Views or opinions, if any, presented in this email are solely those of the
>> author and may not necessarily reflect the views or opinions of redBus.com.
>> Any form of reproduction, dissemination, copying, disclosure,
>> modification,distribution and / or publication of this message without the
>> prior written consent of authorized representative of redbus.
>> <http://redbus.in/>com is strictly prohibited. If you have received this
>> email in error please delete it and notify the sender immediately.Before
>> opening any email and/or attachments, please check them for viruses and
>> other defects.*
>
>
--
*::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachments are confidential and
intended for the named recipient(s) only.E-mail transmission is not
guaranteed to be secure or error-free as information could be intercepted,
corrupted,lost, destroyed, arrive late or incomplete, or may contain
viruses in transmission. The e mail and its contents(with or without
referred errors) shall therefore not attach any liability on the originator
or redBus.com. Views or opinions, if any, presented in this email are
solely those of the author and may not necessarily reflect the views or
opinions of redBus.com. Any form of reproduction, dissemination, copying,
disclosure, modification,distribution and / or publication of this message
without the prior written consent of authorized representative of redbus.
<http://redbus.in/>com is strictly prohibited. If you have received this
email in error please delete it and notify the sender immediately.Before
opening any email and/or attachments, please check them for viruses and
other defects.*
Re: Combine Attributes & Content
Posted by Edward Armes <ed...@gmail.com>.
Hi Sharma,
An alternative could be, duplicate the FlowFile from the HandleHTTPRequest
Processor have one wait while the other goes through an AttributesToJSON
and stores the JSON in a lookup cache. The other FlowFile then grabs that
json from the lookup cache and then use ReplaceText processor mode to add
that specific attribute into the content.
Edward
On Thu, May 21, 2020 at 10:11 AM Dweep Sharma <dw...@redbus.com>
wrote:
> Thanks Matt, UpdateRecord works however, all headers must be added
> manually (unlike AttributesToJson). Since it is one time effort, it will do
> for now.
>
> -Dweep
>
> On Tue, May 19, 2020 at 9:16 PM Matt Burgess <ma...@apache.org> wrote:
>
>> Dweep,
>>
>> Depending on how complex the content JSON is, you might be able to use
>> ReplaceText to smuggle the attributes into the text, but this can be
>> tricky as you need to match on the opening JSON and the rest, and then
>> replace it with the opening JSON, the attributes, then the rest in
>> order to preserve valid JSON.
>>
>> You may also be able to use UpdateRecord to add fields to your
>> content, I believe if the Replacement Value Strategy property is set
>> to "Literal Value" you can use Expression Language which may be able
>> access flow file attributes.
>>
>> An alternative is to use a scripted processor if you're comfortable
>> with a scripting language such as Groovy, as it can "slurp" JSON into
>> a Plain Old Groovy Object (POGO), then you can add fields and
>> serialize back out to JSON. I have a somewhat similar example here
>> [1].
>>
>> Regards,
>> Matt
>>
>> [1]
>> http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
>>
>> On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <dw...@redbus.com>
>> wrote:
>> >
>> > Hi All,
>> >
>> > Does anyone have a strategy to combine Attributes & Content into a
>> single Flowfile?
>> >
>> > AttributesToJson seems like the best way convert all attributes to
>> content -Json but I
>> >
>> > lose the original content.
>> >
>> > Basically, I have a HandleHTTPRequest Processor and headers and body
>> must be combined,
>> >
>> > currently Headers are stored as Attributes in a flowfile
>> >
>> > -Dweep
>> >
>> >
>> > ::DISCLAIMER::
>> >
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>> >
>> > The contents of this e-mail and any attachments are confidential and
>> intended for the named recipient(s) only.E-mail transmission is not
>> guaranteed to be secure or error-free as information could be intercepted,
>> corrupted,lost, destroyed, arrive late or incomplete, or may contain
>> viruses in transmission. The e mail and its contents(with or without
>> referred errors) shall therefore not attach any liability on the originator
>> or redBus.com. Views or opinions, if any, presented in this email are
>> solely those of the author and may not necessarily reflect the views or
>> opinions of redBus.com. Any form of reproduction, dissemination, copying,
>> disclosure, modification,distribution and / or publication of this message
>> without the prior written consent of authorized representative of
>> redbus.com is strictly prohibited. If you have received this email in
>> error please delete it and notify the sender immediately.Before opening any
>> email and/or attachments, please check them for viruses and other defects.
>>
>
>
>
>
> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The
> contents of this e-mail and any attachments are confidential and intended
> for the named recipient(s) only.E-mail transmission is not guaranteed to be
> secure or error-free as information could be intercepted, corrupted,lost,
> destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents(with or without referred errors)
> shall therefore not attach any liability on the originator or redBus.com.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the views or opinions of redBus.com.
> Any form of reproduction, dissemination, copying, disclosure,
> modification,distribution and / or publication of this message without the
> prior written consent of authorized representative of redbus.
> <http://redbus.in/>com is strictly prohibited. If you have received this
> email in error please delete it and notify the sender immediately.Before
> opening any email and/or attachments, please check them for viruses and
> other defects.*
Re: Combine Attributes & Content
Posted by Dweep Sharma <dw...@redbus.com>.
Thanks Matt, UpdateRecord works however, all headers must be added manually
(unlike AttributesToJson). Since it is one time effort, it will do for now.
-Dweep
On Tue, May 19, 2020 at 9:16 PM Matt Burgess <ma...@apache.org> wrote:
> Dweep,
>
> Depending on how complex the content JSON is, you might be able to use
> ReplaceText to smuggle the attributes into the text, but this can be
> tricky as you need to match on the opening JSON and the rest, and then
> replace it with the opening JSON, the attributes, then the rest in
> order to preserve valid JSON.
>
> You may also be able to use UpdateRecord to add fields to your
> content, I believe if the Replacement Value Strategy property is set
> to "Literal Value" you can use Expression Language which may be able
> access flow file attributes.
>
> An alternative is to use a scripted processor if you're comfortable
> with a scripting language such as Groovy, as it can "slurp" JSON into
> a Plain Old Groovy Object (POGO), then you can add fields and
> serialize back out to JSON. I have a somewhat similar example here
> [1].
>
> Regards,
> Matt
>
> [1]
> http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
>
> On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <dw...@redbus.com>
> wrote:
> >
> > Hi All,
> >
> > Does anyone have a strategy to combine Attributes & Content into a
> single Flowfile?
> >
> > AttributesToJson seems like the best way convert all attributes to
> content -Json but I
> >
> > lose the original content.
> >
> > Basically, I have a HandleHTTPRequest Processor and headers and body
> must be combined,
> >
> > currently Headers are stored as Attributes in a flowfile
> >
> > -Dweep
> >
> >
> > ::DISCLAIMER::
> >
> ----------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > The contents of this e-mail and any attachments are confidential and
> intended for the named recipient(s) only.E-mail transmission is not
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted,lost, destroyed, arrive late or incomplete, or may contain
> viruses in transmission. The e mail and its contents(with or without
> referred errors) shall therefore not attach any liability on the originator
> or redBus.com. Views or opinions, if any, presented in this email are
> solely those of the author and may not necessarily reflect the views or
> opinions of redBus.com. Any form of reproduction, dissemination, copying,
> disclosure, modification,distribution and / or publication of this message
> without the prior written consent of authorized representative of
> redbus.com is strictly prohibited. If you have received this email in
> error please delete it and notify the sender immediately.Before opening any
> email and/or attachments, please check them for viruses and other defects.
>
--
*::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachments are confidential and
intended for the named recipient(s) only.E-mail transmission is not
guaranteed to be secure or error-free as information could be intercepted,
corrupted,lost, destroyed, arrive late or incomplete, or may contain
viruses in transmission. The e mail and its contents(with or without
referred errors) shall therefore not attach any liability on the originator
or redBus.com. Views or opinions, if any, presented in this email are
solely those of the author and may not necessarily reflect the views or
opinions of redBus.com. Any form of reproduction, dissemination, copying,
disclosure, modification,distribution and / or publication of this message
without the prior written consent of authorized representative of redbus.
<http://redbus.in/>com is strictly prohibited. If you have received this
email in error please delete it and notify the sender immediately.Before
opening any email and/or attachments, please check them for viruses and
other defects.*
Re: Combine Attributes & Content
Posted by Matt Burgess <ma...@apache.org>.
Dweep,
Depending on how complex the content JSON is, you might be able to use
ReplaceText to smuggle the attributes into the text, but this can be
tricky as you need to match on the opening JSON and the rest, and then
replace it with the opening JSON, the attributes, then the rest in
order to preserve valid JSON.
You may also be able to use UpdateRecord to add fields to your
content, I believe if the Replacement Value Strategy property is set
to "Literal Value" you can use Expression Language which may be able
access flow file attributes.
An alternative is to use a scripted processor if you're comfortable
with a scripting language such as Groovy, as it can "slurp" JSON into
a Plain Old Groovy Object (POGO), then you can add fields and
serialize back out to JSON. I have a somewhat similar example here
[1].
Regards,
Matt
[1] http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <dw...@redbus.com> wrote:
>
> Hi All,
>
> Does anyone have a strategy to combine Attributes & Content into a single Flowfile?
>
> AttributesToJson seems like the best way convert all attributes to content -Json but I
>
> lose the original content.
>
> Basically, I have a HandleHTTPRequest Processor and headers and body must be combined,
>
> currently Headers are stored as Attributes in a flowfile
>
> -Dweep
>
>
> ::DISCLAIMER::
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.