You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Juan Pablo Gardella <ga...@gmail.com> on 2017/11/21 12:10:10 UTC
Extract Record field to an attribute
Hello all,
I'm working with Nifi records. Currently I have a json converted to Avro
object using ConvertRecord processor. I would like to know if it is
possible to use something similar to UpdateAttribute to add an attribute
which evaluates a record expression. Something similar to:
attribute1 -> ${/person/name}
Juan
Re: Extract Record field to an attribute
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Awesome! Thanks Mark, I didn't think multiple records flow file, it makes a
lot of sense your suggestions.
Thanks,
Juan
On Tue, 21 Nov 2017 at 11:29 Mark Payne <ma...@hotmail.com> wrote:
> Juan,
>
> Working with attributes is a little bit trickier in this case because the
> record-oriented processors
> are generally intended to work on 'streams' of records. I.e., each
> FlowFile can have 1 record or
> it can be made up of thousands or more records. So if you have many
> records in a FlowFile, it's
> a little more difficult to extract a field value into an attribute.
>
> So what we do is a little bit different here. We need to group together
> 'like records' into separate
> FlowFiles. For example, if we have 5 records in a FlowFile, and
> /person/name is 'Juan' for the first 2
> and is 'Mark' for the last 3, then we can use PartitionRecord to separate
> our FlowFile into two separate
> FlowFiles, the first containing those records where /person/name is 'Juan'
> and the second FlowFile
> containing those records where /person/name is 'Mark'. Once we have done,
> it now makes more sense
> to extract the name 'Juan' and the name 'Mark' into FlowFile attributes.
> And that's just what PartitionRecord
> does. Each outbound FlowFile will have an attribute that is equal to the
> value of the field specified.
>
> So for example, if you add a single property to PartitionRecord named
> 'person' with a value of /person/name
> and then send in that example FlowFile mentioned above, then you'd get out
> 2 FlowFiles. The first would
> have an attribute 'person' (the name of the property you added is the name
> of the attribute) with a value of
> 'Juan' and the second would have an attribute 'person' with a value of
> 'Mark'.
>
> Also of note - the PartitionRecord processor takes a Record Reader and
> Writer, so this allows you to read
> the data in as JSON and then write it out as Avro. Essentially, it allows
> for an implicit record conversion, so
> you will no longer need your ConvertRecord processor in your flow.
>
> The documentation for PartitionRecord can be found here [1]. If you click
> the 'Additional Details...' link at the
> end of the first paragraph, it will provide quite a bit more documentation
> with examples. Hopefully this all
> makes sense, but if you have further questions, I am happy to elaborate if
> there is something that's not clear.
>
> Thanks!
> -Mark
>
> [1]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.PartitionRecord/index.html
>
>
>
> > On Nov 21, 2017, at 7:10 AM, Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
> >
> > Hello all,
> >
> > I'm working with Nifi records. Currently I have a json converted to Avro
> object using ConvertRecord processor. I would like to know if it is
> possible to use something similar to UpdateAttribute to add an attribute
> which evaluates a record expression. Something similar to:
> >
> > attribute1 -> ${/person/name}
> >
> > Juan
>
>
Re: Extract Record field to an attribute
Posted by Mark Payne <ma...@hotmail.com>.
Juan,
Working with attributes is a little bit trickier in this case because the record-oriented processors
are generally intended to work on 'streams' of records. I.e., each FlowFile can have 1 record or
it can be made up of thousands or more records. So if you have many records in a FlowFile, it's
a little more difficult to extract a field value into an attribute.
So what we do is a little bit different here. We need to group together 'like records' into separate
FlowFiles. For example, if we have 5 records in a FlowFile, and /person/name is 'Juan' for the first 2
and is 'Mark' for the last 3, then we can use PartitionRecord to separate our FlowFile into two separate
FlowFiles, the first containing those records where /person/name is 'Juan' and the second FlowFile
containing those records where /person/name is 'Mark'. Once we have done, it now makes more sense
to extract the name 'Juan' and the name 'Mark' into FlowFile attributes. And that's just what PartitionRecord
does. Each outbound FlowFile will have an attribute that is equal to the value of the field specified.
So for example, if you add a single property to PartitionRecord named 'person' with a value of /person/name
and then send in that example FlowFile mentioned above, then you'd get out 2 FlowFiles. The first would
have an attribute 'person' (the name of the property you added is the name of the attribute) with a value of
'Juan' and the second would have an attribute 'person' with a value of 'Mark'.
Also of note - the PartitionRecord processor takes a Record Reader and Writer, so this allows you to read
the data in as JSON and then write it out as Avro. Essentially, it allows for an implicit record conversion, so
you will no longer need your ConvertRecord processor in your flow.
The documentation for PartitionRecord can be found here [1]. If you click the 'Additional Details...' link at the
end of the first paragraph, it will provide quite a bit more documentation with examples. Hopefully this all
makes sense, but if you have further questions, I am happy to elaborate if there is something that's not clear.
Thanks!
-Mark
[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.PartitionRecord/index.html
> On Nov 21, 2017, at 7:10 AM, Juan Pablo Gardella <ga...@gmail.com> wrote:
>
> Hello all,
>
> I'm working with Nifi records. Currently I have a json converted to Avro object using ConvertRecord processor. I would like to know if it is possible to use something similar to UpdateAttribute to add an attribute which evaluates a record expression. Something similar to:
>
> attribute1 -> ${/person/name}
>
> Juan