You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by "Tholen, Sue (Chief Data Office)" <Su...@thehartford.com> on 2020/04/13 17:24:33 UTC

Infer Avro Schema [CONFIDENTIAL]

I thought that version management ensured backward compatibility but cannot find an equivalent function in v1.10 for "infer avro schema".
Nor could I find anything in the release notes about it going away...
Does a similar capability exist?
************************************************************
This communication, including attachments, is for the exclusive use of addressee and may contain proprietary, confidential and/or privileged information.  If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited.  If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this communication and destroy all copies.
************************************************************

Re: Infer Avro Schema [CONFIDENTIAL]

Posted by Joe Witt <jo...@gmail.com>.
Yep definitely agree Chris.  This is a good time to switch to using record
readers and writers and the built in schema handling.

Thanks

On Tue, Apr 14, 2020 at 3:06 AM Chris Sampson
<ch...@naimuri.com.invalid> wrote:

> An alternative (without adding the kite nar back into your installation),
> is to use one of the "standard" Record processors (e.g. ConvertRecord,
> UpdateRecord, ValidateRecord or CalculateRecordStats) with an appropriate
> RecordReader with schema access set to "Infer Schema" and schema output to
> "Write as Attribute" (I'm paraphrasing the settings from memory, but they
> should be fairly evident). The RecordWriter can then output in the same
> format as the reader and "Inherit Schema", so no need to define an explicit
> schema anywhere.
>
> This has the added advantage of adding in the opportunity to conduct other
> business logic at the same time, e.g. confirming the data you're processing
> is indeed json or counting the number of records in a flowfile, etc.
>
> I couldn't give much of a statement about the comparative
> accuracy/efficiency of the different approaches though in afraid.
>
>
> Cheers,
>
> Chris Sampson
>
> On Mon, 13 Apr 2020, 18:34 Joe Witt, <jo...@gmail.com> wrote:
>
> > Sue,
> >
> > Due to space issues we had to remove certain nars from the convenience
> > binary we ship.  They're listed here
> > https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance. We
> > still do publish the nars and you can use them but you have to add them
> to
> > your install.
> >
> > Thanks
> > Joe
> >
> > On Mon, Apr 13, 2020 at 1:26 PM Tholen, Sue (Chief Data Office) <
> > Susan.Tholen@thehartford.com> wrote:
> >
> > > I thought that version management ensured backward compatibility but
> > > cannot find an equivalent function in v1.10 for "infer avro schema".
> > > Nor could I find anything in the release notes about it going away...
> > > Does a similar capability exist?
> > > ************************************************************
> > > This communication, including attachments, is for the exclusive use of
> > > addressee and may contain proprietary, confidential and/or privileged
> > > information.  If you are not the intended recipient, any use, copying,
> > > disclosure, dissemination or distribution is strictly prohibited.  If
> you
> > > are not the intended recipient, please notify the sender immediately by
> > > return e-mail, delete this communication and destroy all copies.
> > > ************************************************************
> > >
> >
>

Re: Infer Avro Schema [CONFIDENTIAL]

Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
An alternative (without adding the kite nar back into your installation),
is to use one of the "standard" Record processors (e.g. ConvertRecord,
UpdateRecord, ValidateRecord or CalculateRecordStats) with an appropriate
RecordReader with schema access set to "Infer Schema" and schema output to
"Write as Attribute" (I'm paraphrasing the settings from memory, but they
should be fairly evident). The RecordWriter can then output in the same
format as the reader and "Inherit Schema", so no need to define an explicit
schema anywhere.

This has the added advantage of adding in the opportunity to conduct other
business logic at the same time, e.g. confirming the data you're processing
is indeed json or counting the number of records in a flowfile, etc.

I couldn't give much of a statement about the comparative
accuracy/efficiency of the different approaches though in afraid.


Cheers,

Chris Sampson

On Mon, 13 Apr 2020, 18:34 Joe Witt, <jo...@gmail.com> wrote:

> Sue,
>
> Due to space issues we had to remove certain nars from the convenience
> binary we ship.  They're listed here
> https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance. We
> still do publish the nars and you can use them but you have to add them to
> your install.
>
> Thanks
> Joe
>
> On Mon, Apr 13, 2020 at 1:26 PM Tholen, Sue (Chief Data Office) <
> Susan.Tholen@thehartford.com> wrote:
>
> > I thought that version management ensured backward compatibility but
> > cannot find an equivalent function in v1.10 for "infer avro schema".
> > Nor could I find anything in the release notes about it going away...
> > Does a similar capability exist?
> > ************************************************************
> > This communication, including attachments, is for the exclusive use of
> > addressee and may contain proprietary, confidential and/or privileged
> > information.  If you are not the intended recipient, any use, copying,
> > disclosure, dissemination or distribution is strictly prohibited.  If you
> > are not the intended recipient, please notify the sender immediately by
> > return e-mail, delete this communication and destroy all copies.
> > ************************************************************
> >
>

Re: Infer Avro Schema [CONFIDENTIAL]

Posted by Joe Witt <jo...@gmail.com>.
Sue,

Due to space issues we had to remove certain nars from the convenience
binary we ship.  They're listed here
https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance. We
still do publish the nars and you can use them but you have to add them to
your install.

Thanks
Joe

On Mon, Apr 13, 2020 at 1:26 PM Tholen, Sue (Chief Data Office) <
Susan.Tholen@thehartford.com> wrote:

> I thought that version management ensured backward compatibility but
> cannot find an equivalent function in v1.10 for "infer avro schema".
> Nor could I find anything in the release notes about it going away...
> Does a similar capability exist?
> ************************************************************
> This communication, including attachments, is for the exclusive use of
> addressee and may contain proprietary, confidential and/or privileged
> information.  If you are not the intended recipient, any use, copying,
> disclosure, dissemination or distribution is strictly prohibited.  If you
> are not the intended recipient, please notify the sender immediately by
> return e-mail, delete this communication and destroy all copies.
> ************************************************************
>