You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jairo Henao <ja...@gmail.com> on 2020/04/08 18:01:09 UTC
How to read a schema from a ScriptedReader
Hi all,
Is there a way from a ScriptedReader (Controller Service) to read a schema
that is stored in an AvroSchemaRegistry?
I am using Groovy as a language.
Additionally, how can I read additional properties that I have added to my
ControllerService from Groovy code.
Thanks in advance
--
Saludos
Jairo Henao
Re: How to read a schema from a ScriptedReader
Posted by Jairo Henao <ja...@gmail.com>.
Thanks Matt,
Yes, I am subscribed to the development list.
I will review your comments. At the moment I am adding the schemas as
additional properties to the ScriptedReader.
I will try what you mention to access SchemaRegistry and send you some
comments.
On Wed, Apr 8, 2020 at 10:30 PM Matt Burgess <ma...@apache.org> wrote:
> As a follow up, while implementing [1] I ran into [2] which is a
> larger issue but affects the RecordSetWriters more than Readers as
> they add more properties that are expected to come (in our case) from
> the script. That makes things a bit more complicated, but I am still
> looking into it. If you are coding a ScriptedReader you might be ok
> with the guidance I gave earlier, but I have my doubts that trying to
> use a ScriptedRecordSetWriter with a SchemaRegistry will work right
> now. I'll post any findings in either/both [1] and [2], and as always
> I welcome all comments, questions, and suggestions :)
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-7343
> [2] https://issues.apache.org/jira/browse/NIFI-5115
>
> On Wed, Apr 8, 2020 at 8:19 PM Matt Burgess <ma...@apache.org> wrote:
> >
> > Jairo,
> > We should probably move this to the dev list since we're getting into
> > the NiFi API, but I wasn't sure you were subscribed, if you are let me
> > know and we can move it over there.
> >
> > StandardConfigurationContext is in framework-core so you should not
> > count on being able to use that class per se. Instead you should add a
> > property like the one in SchemaAccessUtils:
> >
> > public static final PropertyDescriptor SCHEMA_REGISTRY = new
> > PropertyDescriptor.Builder()
> > .name("schema-registry")
> > .displayName("Schema Registry")
> > .description("Specifies the Controller Service to use for the
> > Schema Registry")
> > .identifiesControllerService(SchemaRegistry.class)
> > .required(false)
> > .build();
> >
> > Then from your code you can add that property in your overridden
> > getSupportedPropertyDescriptors() method, and when you need it you can
> > get the property from the context, but don't refer to it as a
> > StandardConfigurationContext, there's an interface that has the same
> > methods and is in the API that you have access to already:
> >
> > final SchemaRegistry schemaRegistry =
> >
> context.getProperty(SCHEMA_REGISTRY).asControllerService(SchemaRegistry.class)
> >
> > Note that we're not getting the registry directly from the context,
> > rather we're getting the property value from the context and then
> > finding the corresponding SchemaRegistry, I believe that's done "under
> > the hood" by the ControllerServiceLookup, the
> > ".asControllerService(SchemaRegistry.class)" is the key part there.
> >
> > From there you can use the SchemaRegistry interface methods to get at
> > the other things, for example:
> >
> > descriptor = getPropertyDescriptor(SCHEMA_ACCESS_STRATEGY.getName());
> > schemaAccess = context.getProperty(descriptor).getValue();
> > schemaAccessStrategy = getSchemaAccessStrategy(schemaAccess,
> > schemaRegistry, context);
> >
> > Note that last method getSchemaAccessStrategy() is from
> > SchemaAccessUtils which you don't currently have access to, so you
> > might want to just copy all that over to your script. If you follow
> > the calls and code from SchemaRegistryService, you should be able to
> > bring any/all of that into your ScriptedReader so it can do the same
> > kind of processing.
> >
> > Regards,
> > Matt
> >
> > On Wed, Apr 8, 2020 at 7:03 PM Jairo Henao <ja...@gmail.com>
> wrote:
> > >
> > > Thanks Matt,
> > >
> > > I have made some changes using AvroReader as an example, but I am
> still able to access other ControllerServices.
> > >
> > > From the RecordReaderFactory I have access to a context that is of
> type org.apache.nifi.controller.service.StandardConfigurationContext but
> with this I don't know how to access the AvroSchemaRegistry.
> > >
> > > Do you know how I can access the ControllerServiceLookup?
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess <ma...@apache.org>
> wrote:
> > >>
> > >> Jairo,
> > >>
> > >> It is possible for a ScriptedReader to use a SchemaRegistry, although
> > >> it isn't currently as easy as it could be. When the Scripted
> > >> Reader/Writer components were added [1], much of the reusable code for
> > >> schema registry processing was in the service implementation NAR, so
> > >> the aforementioned Jira explains that using the SchemaRegistry
> > >> interface is certainly possible in ScriptedReader but fairly
> > >> cumbersome. Since then the SchemaRegistryService and associated util
> > >> classes were moved to a nifi-avro-record-utils [2] so
> > >> extensions/components can leverage schema registries. Unfortunately
> > >> nifi-avro-record-utils is not currently included in the scripting NAR,
> > >> but I have written up [3] to add this.
> > >>
> > >> In the meantime, take a look at SchemaRegistryService,
> > >> SchemaAccessUtils, and an implementation class such as AvroReader, you
> > >> could copy and paste all the necessary code to get your ScriptedReader
> > >> to interact with a schema registry.
> > >>
> > >> Regards,
> > >> Matt
> > >>
> > >> [1] https://issues.apache.org/jira/browse/NIFI-3734
> > >> [2] https://issues.apache.org/jira/browse/NIFI-5123
> > >> [3] https://issues.apache.org/jira/browse/NIFI-7343
> > >>
> > >> On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao <ja...@gmail.com>
> wrote:
> > >> >
> > >> > Hi all,
> > >> >
> > >> > Is there a way from a ScriptedReader (Controller Service) to read a
> schema that is stored in an AvroSchemaRegistry?
> > >> >
> > >> > I am using Groovy as a language.
> > >> >
> > >> > Additionally, how can I read additional properties that I have
> added to my ControllerService from Groovy code.
> > >> >
> > >> > Thanks in advance
> > >> >
> > >> >
> > >> > --
> > >> > Saludos
> > >> >
> > >> > Jairo Henao
> > >
> > >
> > >
> > > --
> > > Saludos
> > >
> > > Jairo Henao
> > >
> > > Chat Skype: jairo.henao.05
> > >
>
--
Saludos
Jairo Henao
*Chat Skype: jairo.henao.05*
Re: How to read a schema from a ScriptedReader
Posted by Matt Burgess <ma...@apache.org>.
As a follow up, while implementing [1] I ran into [2] which is a
larger issue but affects the RecordSetWriters more than Readers as
they add more properties that are expected to come (in our case) from
the script. That makes things a bit more complicated, but I am still
looking into it. If you are coding a ScriptedReader you might be ok
with the guidance I gave earlier, but I have my doubts that trying to
use a ScriptedRecordSetWriter with a SchemaRegistry will work right
now. I'll post any findings in either/both [1] and [2], and as always
I welcome all comments, questions, and suggestions :)
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-7343
[2] https://issues.apache.org/jira/browse/NIFI-5115
On Wed, Apr 8, 2020 at 8:19 PM Matt Burgess <ma...@apache.org> wrote:
>
> Jairo,
> We should probably move this to the dev list since we're getting into
> the NiFi API, but I wasn't sure you were subscribed, if you are let me
> know and we can move it over there.
>
> StandardConfigurationContext is in framework-core so you should not
> count on being able to use that class per se. Instead you should add a
> property like the one in SchemaAccessUtils:
>
> public static final PropertyDescriptor SCHEMA_REGISTRY = new
> PropertyDescriptor.Builder()
> .name("schema-registry")
> .displayName("Schema Registry")
> .description("Specifies the Controller Service to use for the
> Schema Registry")
> .identifiesControllerService(SchemaRegistry.class)
> .required(false)
> .build();
>
> Then from your code you can add that property in your overridden
> getSupportedPropertyDescriptors() method, and when you need it you can
> get the property from the context, but don't refer to it as a
> StandardConfigurationContext, there's an interface that has the same
> methods and is in the API that you have access to already:
>
> final SchemaRegistry schemaRegistry =
> context.getProperty(SCHEMA_REGISTRY).asControllerService(SchemaRegistry.class)
>
> Note that we're not getting the registry directly from the context,
> rather we're getting the property value from the context and then
> finding the corresponding SchemaRegistry, I believe that's done "under
> the hood" by the ControllerServiceLookup, the
> ".asControllerService(SchemaRegistry.class)" is the key part there.
>
> From there you can use the SchemaRegistry interface methods to get at
> the other things, for example:
>
> descriptor = getPropertyDescriptor(SCHEMA_ACCESS_STRATEGY.getName());
> schemaAccess = context.getProperty(descriptor).getValue();
> schemaAccessStrategy = getSchemaAccessStrategy(schemaAccess,
> schemaRegistry, context);
>
> Note that last method getSchemaAccessStrategy() is from
> SchemaAccessUtils which you don't currently have access to, so you
> might want to just copy all that over to your script. If you follow
> the calls and code from SchemaRegistryService, you should be able to
> bring any/all of that into your ScriptedReader so it can do the same
> kind of processing.
>
> Regards,
> Matt
>
> On Wed, Apr 8, 2020 at 7:03 PM Jairo Henao <ja...@gmail.com> wrote:
> >
> > Thanks Matt,
> >
> > I have made some changes using AvroReader as an example, but I am still able to access other ControllerServices.
> >
> > From the RecordReaderFactory I have access to a context that is of type org.apache.nifi.controller.service.StandardConfigurationContext but with this I don't know how to access the AvroSchemaRegistry.
> >
> > Do you know how I can access the ControllerServiceLookup?
> >
> >
> >
> >
> >
> >
> > On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess <ma...@apache.org> wrote:
> >>
> >> Jairo,
> >>
> >> It is possible for a ScriptedReader to use a SchemaRegistry, although
> >> it isn't currently as easy as it could be. When the Scripted
> >> Reader/Writer components were added [1], much of the reusable code for
> >> schema registry processing was in the service implementation NAR, so
> >> the aforementioned Jira explains that using the SchemaRegistry
> >> interface is certainly possible in ScriptedReader but fairly
> >> cumbersome. Since then the SchemaRegistryService and associated util
> >> classes were moved to a nifi-avro-record-utils [2] so
> >> extensions/components can leverage schema registries. Unfortunately
> >> nifi-avro-record-utils is not currently included in the scripting NAR,
> >> but I have written up [3] to add this.
> >>
> >> In the meantime, take a look at SchemaRegistryService,
> >> SchemaAccessUtils, and an implementation class such as AvroReader, you
> >> could copy and paste all the necessary code to get your ScriptedReader
> >> to interact with a schema registry.
> >>
> >> Regards,
> >> Matt
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-3734
> >> [2] https://issues.apache.org/jira/browse/NIFI-5123
> >> [3] https://issues.apache.org/jira/browse/NIFI-7343
> >>
> >> On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao <ja...@gmail.com> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > Is there a way from a ScriptedReader (Controller Service) to read a schema that is stored in an AvroSchemaRegistry?
> >> >
> >> > I am using Groovy as a language.
> >> >
> >> > Additionally, how can I read additional properties that I have added to my ControllerService from Groovy code.
> >> >
> >> > Thanks in advance
> >> >
> >> >
> >> > --
> >> > Saludos
> >> >
> >> > Jairo Henao
> >
> >
> >
> > --
> > Saludos
> >
> > Jairo Henao
> >
> > Chat Skype: jairo.henao.05
> >
Re: How to read a schema from a ScriptedReader
Posted by Matt Burgess <ma...@apache.org>.
Jairo,
We should probably move this to the dev list since we're getting into
the NiFi API, but I wasn't sure you were subscribed, if you are let me
know and we can move it over there.
StandardConfigurationContext is in framework-core so you should not
count on being able to use that class per se. Instead you should add a
property like the one in SchemaAccessUtils:
public static final PropertyDescriptor SCHEMA_REGISTRY = new
PropertyDescriptor.Builder()
.name("schema-registry")
.displayName("Schema Registry")
.description("Specifies the Controller Service to use for the
Schema Registry")
.identifiesControllerService(SchemaRegistry.class)
.required(false)
.build();
Then from your code you can add that property in your overridden
getSupportedPropertyDescriptors() method, and when you need it you can
get the property from the context, but don't refer to it as a
StandardConfigurationContext, there's an interface that has the same
methods and is in the API that you have access to already:
final SchemaRegistry schemaRegistry =
context.getProperty(SCHEMA_REGISTRY).asControllerService(SchemaRegistry.class)
Note that we're not getting the registry directly from the context,
rather we're getting the property value from the context and then
finding the corresponding SchemaRegistry, I believe that's done "under
the hood" by the ControllerServiceLookup, the
".asControllerService(SchemaRegistry.class)" is the key part there.
From there you can use the SchemaRegistry interface methods to get at
the other things, for example:
descriptor = getPropertyDescriptor(SCHEMA_ACCESS_STRATEGY.getName());
schemaAccess = context.getProperty(descriptor).getValue();
schemaAccessStrategy = getSchemaAccessStrategy(schemaAccess,
schemaRegistry, context);
Note that last method getSchemaAccessStrategy() is from
SchemaAccessUtils which you don't currently have access to, so you
might want to just copy all that over to your script. If you follow
the calls and code from SchemaRegistryService, you should be able to
bring any/all of that into your ScriptedReader so it can do the same
kind of processing.
Regards,
Matt
On Wed, Apr 8, 2020 at 7:03 PM Jairo Henao <ja...@gmail.com> wrote:
>
> Thanks Matt,
>
> I have made some changes using AvroReader as an example, but I am still able to access other ControllerServices.
>
> From the RecordReaderFactory I have access to a context that is of type org.apache.nifi.controller.service.StandardConfigurationContext but with this I don't know how to access the AvroSchemaRegistry.
>
> Do you know how I can access the ControllerServiceLookup?
>
>
>
>
>
>
> On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess <ma...@apache.org> wrote:
>>
>> Jairo,
>>
>> It is possible for a ScriptedReader to use a SchemaRegistry, although
>> it isn't currently as easy as it could be. When the Scripted
>> Reader/Writer components were added [1], much of the reusable code for
>> schema registry processing was in the service implementation NAR, so
>> the aforementioned Jira explains that using the SchemaRegistry
>> interface is certainly possible in ScriptedReader but fairly
>> cumbersome. Since then the SchemaRegistryService and associated util
>> classes were moved to a nifi-avro-record-utils [2] so
>> extensions/components can leverage schema registries. Unfortunately
>> nifi-avro-record-utils is not currently included in the scripting NAR,
>> but I have written up [3] to add this.
>>
>> In the meantime, take a look at SchemaRegistryService,
>> SchemaAccessUtils, and an implementation class such as AvroReader, you
>> could copy and paste all the necessary code to get your ScriptedReader
>> to interact with a schema registry.
>>
>> Regards,
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3734
>> [2] https://issues.apache.org/jira/browse/NIFI-5123
>> [3] https://issues.apache.org/jira/browse/NIFI-7343
>>
>> On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao <ja...@gmail.com> wrote:
>> >
>> > Hi all,
>> >
>> > Is there a way from a ScriptedReader (Controller Service) to read a schema that is stored in an AvroSchemaRegistry?
>> >
>> > I am using Groovy as a language.
>> >
>> > Additionally, how can I read additional properties that I have added to my ControllerService from Groovy code.
>> >
>> > Thanks in advance
>> >
>> >
>> > --
>> > Saludos
>> >
>> > Jairo Henao
>
>
>
> --
> Saludos
>
> Jairo Henao
>
> Chat Skype: jairo.henao.05
>
Re: How to read a schema from a ScriptedReader
Posted by Jairo Henao <ja...@gmail.com>.
Thanks Matt,
I have made some changes using AvroReader as an example, but I am still
able to access other ControllerServices.
From the RecordReaderFactory I have access to a context that is of type
*org.apache.nifi.controller.service.StandardConfigurationContext* but with
this I don't know how to access the AvroSchemaRegistry.
Do you know how I can access the ControllerServiceLookup?
On Wed, Apr 8, 2020 at 3:19 PM Matt Burgess <ma...@apache.org> wrote:
> Jairo,
>
> It is possible for a ScriptedReader to use a SchemaRegistry, although
> it isn't currently as easy as it could be. When the Scripted
> Reader/Writer components were added [1], much of the reusable code for
> schema registry processing was in the service implementation NAR, so
> the aforementioned Jira explains that using the SchemaRegistry
> interface is certainly possible in ScriptedReader but fairly
> cumbersome. Since then the SchemaRegistryService and associated util
> classes were moved to a nifi-avro-record-utils [2] so
> extensions/components can leverage schema registries. Unfortunately
> nifi-avro-record-utils is not currently included in the scripting NAR,
> but I have written up [3] to add this.
>
> In the meantime, take a look at SchemaRegistryService,
> SchemaAccessUtils, and an implementation class such as AvroReader, you
> could copy and paste all the necessary code to get your ScriptedReader
> to interact with a schema registry.
>
> Regards,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-3734
> [2] https://issues.apache.org/jira/browse/NIFI-5123
> [3] https://issues.apache.org/jira/browse/NIFI-7343
>
> On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao <ja...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > Is there a way from a ScriptedReader (Controller Service) to read a
> schema that is stored in an AvroSchemaRegistry?
> >
> > I am using Groovy as a language.
> >
> > Additionally, how can I read additional properties that I have added to
> my ControllerService from Groovy code.
> >
> > Thanks in advance
> >
> >
> > --
> > Saludos
> >
> > Jairo Henao
>
--
Saludos
Jairo Henao
*Chat Skype: jairo.henao.05*
Re: How to read a schema from a ScriptedReader
Posted by Matt Burgess <ma...@apache.org>.
Jairo,
It is possible for a ScriptedReader to use a SchemaRegistry, although
it isn't currently as easy as it could be. When the Scripted
Reader/Writer components were added [1], much of the reusable code for
schema registry processing was in the service implementation NAR, so
the aforementioned Jira explains that using the SchemaRegistry
interface is certainly possible in ScriptedReader but fairly
cumbersome. Since then the SchemaRegistryService and associated util
classes were moved to a nifi-avro-record-utils [2] so
extensions/components can leverage schema registries. Unfortunately
nifi-avro-record-utils is not currently included in the scripting NAR,
but I have written up [3] to add this.
In the meantime, take a look at SchemaRegistryService,
SchemaAccessUtils, and an implementation class such as AvroReader, you
could copy and paste all the necessary code to get your ScriptedReader
to interact with a schema registry.
Regards,
Matt
[1] https://issues.apache.org/jira/browse/NIFI-3734
[2] https://issues.apache.org/jira/browse/NIFI-5123
[3] https://issues.apache.org/jira/browse/NIFI-7343
On Wed, Apr 8, 2020 at 2:01 PM Jairo Henao <ja...@gmail.com> wrote:
>
> Hi all,
>
> Is there a way from a ScriptedReader (Controller Service) to read a schema that is stored in an AvroSchemaRegistry?
>
> I am using Groovy as a language.
>
> Additionally, how can I read additional properties that I have added to my ControllerService from Groovy code.
>
> Thanks in advance
>
>
> --
> Saludos
>
> Jairo Henao