You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Austin Tao via users <us...@nifi.apache.org> on 2022/12/16 20:45:41 UTC
ASF Jira account creation, PublishKafkaRecord issue
To whom it may concern,
Hello, I would like to report an issue for Nifi. But, following the new Jira Guidelines, I would therefore like to request that an account for ASF Jira in order to create a ticket.
In regards to the bug, using Nifi 1.19.1 I would like to send a tombstone message (null payload) to Kafka and using the Confluent JDBC sink connector (with delete.enabled=true) to delete a record in our Postgres database. I believe as of Nifi 1.19, PublishKafkaRecord_2_6 now supports the 'Publish Strategy: Use Wrapper' feature which allows setting the Kafka message key and value (Primary Key as the Kafka key, null for the Kafka value). For the Record Key Writer, I'm using an AvroRecordSetWriter to validate and serialize the key against the confluent schema registry (Schema Write Strategy: Confluent Schema Registry Reference, Schema Access Strategy: Use 'Schema Name' Property) but sending the message I come across the error:PublishKafkaRecord_2_6[id=XXX] Failed to send FlowFile[filename=XXX] to Kafka: org.apache.nifi.processor.exception.ProcessException: Could not determine the Avro Schema to use for writing the content
- Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Cannot write Confluent Schema Registry Reference because the Schema Identifier is not known
I can confirm the configuration for the for the AvroRecordSetWriter, ConfluentSchemaRegistry controllers, and PublishKafkaRecord processor are all configured correctly, as I can send the Kafka message just fine using the default Publish Strategy (Use Content as Record Value). It only occurs using Use Wrapper, and the ConfluentSchemaRegistry.
A workaround that has worked was for using JsonRecordSetWriter w/ embedded JSON schemas, but it would be nice to continue using our Avro Schema Registry for this.
I'd appreciate if someone had any advice or experience with this issue, otherwise I'd like to log an issue in JIRA.
Thank you,Austin Tao
Re: ASF Jira account creation, PublishKafkaRecord issue
Posted by Paul Grey <gr...@gmail.com>.
Hi Austin.
Thanks so much for the detail in your initial report. I was able to
isolate an issue with the previous work. I've filed NIFI-10993 to expedite
things, and am working a fix.
My understanding of the current guidance for account creation is that you
would email private@nifi.apache.org, as described here [1]. Your
contributions are definitely welcome, and hopefully this step isn't too
much of a bump in the road.
[1] https://infra.apache.org/jira-guidelines.html#who
On Fri, Dec 16, 2022 at 8:00 PM Paul Grey <gr...@gmail.com> wrote:
> Thanks for that. Still digging; stay tuned...
>
> On Fri, Dec 16, 2022 at 4:32 PM Austin Tao <ta...@yahoo.ca> wrote:
>
>> Hi Paul,
>>
>> Thank you for your quick reply. Keeping some privacy, attached is the
>> configuration used for the processor, and the "AvroRecordSetWriter - key"
>> I'm using.
>>
>> On Friday, December 16, 2022 at 03:58:10 p.m. EST, Paul Grey <
>> greyp95@gmail.com> wrote:
>>
>>
>> Hi. I worked that feature; I'll have a look.
>>
>> When the "Publish Strategy" is set to "Use Wrapper", my recollection is
>> that a second record schema is needed for the key, depending on the
>> processor configuration as a whole. Are you able to share that processor
>> configuration?
>>
>> Thanks
>>
>> On Fri, Dec 16, 2022 at 3:46 PM Austin Tao via users <
>> users@nifi.apache.org> wrote:
>>
>> To whom it may concern,
>>
>> Hello, I would like to report an issue for Nifi. But, following the new
>> Jira Guidelines, I would therefore like to request that an account for ASF
>> Jira in order to create a ticket.
>>
>> In regards to the bug, using Nifi 1.19.1 I would like to send a tombstone
>> message (null payload) to Kafka and using the Confluent JDBC sink
>> connector
>> <https://docs.confluent.io/kafka-connectors/jdbc/current/sink-connector/sink_config_options.html> (with
>> delete.enabled=true) to delete a record in our Postgres database. I believe
>> as of Nifi 1.19, PublishKafkaRecord_2_6 now supports the '*Publish
>> Strategy*: Use Wrapper' feature which allows setting the Kafka message
>> key and value (Primary Key as the Kafka key, null for the Kafka value). For
>> the Record Key Writer, I'm using an AvroRecordSetWriter to validate and
>> serialize the key against the confluent schema registry (*Schema Write
>> Strategy*: Confluent Schema Registry Reference, *Schema Access Strategy*:
>> Use 'Schema Name' Property) but sending the message I come across the error:
>> *PublishKafkaRecord_2_6[id=XXX] Failed to send FlowFile[filename=XXX] to
>> Kafka: org.apache.nifi.processor.exception.ProcessException: Could not
>> determine the Avro Schema to use for writing the content*
>>
>> *- Caused by: org.apache.nifi.schema.access.SchemaNotFoundException:
>> Cannot write Confluent Schema Registry Reference because the Schema
>> Identifier is not known*
>>
>>
>> I can confirm the configuration for the for the AvroRecordSetWriter,
>> ConfluentSchemaRegistry controllers, and PublishKafkaRecord processor are
>> all configured correctly, as I can send the Kafka message just fine using
>> the default *Publish Strategy* (Use Content as Record Value). It only
>> occurs using Use Wrapper, and the ConfluentSchemaRegistry.
>>
>> A workaround that has worked was for using JsonRecordSetWriter w/
>> embedded JSON schemas, but it would be nice to continue using our Avro
>> Schema Registry for this.
>>
>> I'd appreciate if someone had any advice or experience with this issue,
>> otherwise I'd like to log an issue in JIRA.
>>
>> Thank you,
>> Austin Tao
>>
>>
Re: ASF Jira account creation, PublishKafkaRecord issue
Posted by Paul Grey <gr...@gmail.com>.
Thanks for that. Still digging; stay tuned...
On Fri, Dec 16, 2022 at 4:32 PM Austin Tao <ta...@yahoo.ca> wrote:
> Hi Paul,
>
> Thank you for your quick reply. Keeping some privacy, attached is the
> configuration used for the processor, and the "AvroRecordSetWriter - key"
> I'm using.
>
> On Friday, December 16, 2022 at 03:58:10 p.m. EST, Paul Grey <
> greyp95@gmail.com> wrote:
>
>
> Hi. I worked that feature; I'll have a look.
>
> When the "Publish Strategy" is set to "Use Wrapper", my recollection is
> that a second record schema is needed for the key, depending on the
> processor configuration as a whole. Are you able to share that processor
> configuration?
>
> Thanks
>
> On Fri, Dec 16, 2022 at 3:46 PM Austin Tao via users <
> users@nifi.apache.org> wrote:
>
> To whom it may concern,
>
> Hello, I would like to report an issue for Nifi. But, following the new
> Jira Guidelines, I would therefore like to request that an account for ASF
> Jira in order to create a ticket.
>
> In regards to the bug, using Nifi 1.19.1 I would like to send a tombstone
> message (null payload) to Kafka and using the Confluent JDBC sink
> connector
> <https://docs.confluent.io/kafka-connectors/jdbc/current/sink-connector/sink_config_options.html> (with
> delete.enabled=true) to delete a record in our Postgres database. I believe
> as of Nifi 1.19, PublishKafkaRecord_2_6 now supports the '*Publish
> Strategy*: Use Wrapper' feature which allows setting the Kafka message
> key and value (Primary Key as the Kafka key, null for the Kafka value). For
> the Record Key Writer, I'm using an AvroRecordSetWriter to validate and
> serialize the key against the confluent schema registry (*Schema Write
> Strategy*: Confluent Schema Registry Reference, *Schema Access Strategy*:
> Use 'Schema Name' Property) but sending the message I come across the error:
> *PublishKafkaRecord_2_6[id=XXX] Failed to send FlowFile[filename=XXX] to
> Kafka: org.apache.nifi.processor.exception.ProcessException: Could not
> determine the Avro Schema to use for writing the content*
>
> *- Caused by: org.apache.nifi.schema.access.SchemaNotFoundException:
> Cannot write Confluent Schema Registry Reference because the Schema
> Identifier is not known*
>
>
> I can confirm the configuration for the for the AvroRecordSetWriter,
> ConfluentSchemaRegistry controllers, and PublishKafkaRecord processor are
> all configured correctly, as I can send the Kafka message just fine using
> the default *Publish Strategy* (Use Content as Record Value). It only
> occurs using Use Wrapper, and the ConfluentSchemaRegistry.
>
> A workaround that has worked was for using JsonRecordSetWriter w/ embedded
> JSON schemas, but it would be nice to continue using our Avro Schema
> Registry for this.
>
> I'd appreciate if someone had any advice or experience with this issue,
> otherwise I'd like to log an issue in JIRA.
>
> Thank you,
> Austin Tao
>
>
Re: ASF Jira account creation, PublishKafkaRecord issue
Posted by Austin Tao via users <us...@nifi.apache.org>.
Hi Paul,
Thank you for your quick reply. Keeping some privacy, attached is the configuration used for the processor, and the "AvroRecordSetWriter - key" I'm using.
On Friday, December 16, 2022 at 03:58:10 p.m. EST, Paul Grey <gr...@gmail.com> wrote:
Hi. I worked that feature; I'll have a look.
When the "Publish Strategy" is set to "Use Wrapper", my recollection is that a second record schema is needed for the key, depending on the processor configuration as a whole. Are you able to share that processor configuration?
Thanks
On Fri, Dec 16, 2022 at 3:46 PM Austin Tao via users <us...@nifi.apache.org> wrote:
To whom it may concern,
Hello, I would like to report an issue for Nifi. But, following the new Jira Guidelines, I would therefore like to request that an account for ASF Jira in order to create a ticket.
In regards to the bug, using Nifi 1.19.1 I would like to send a tombstone message (null payload) to Kafka and using the Confluent JDBC sink connector (with delete.enabled=true) to delete a record in our Postgres database. I believe as of Nifi 1.19, PublishKafkaRecord_2_6 now supports the 'Publish Strategy: Use Wrapper' feature which allows setting the Kafka message key and value (Primary Key as the Kafka key, null for the Kafka value). For the Record Key Writer, I'm using an AvroRecordSetWriter to validate and serialize the key against the confluent schema registry (Schema Write Strategy: Confluent Schema Registry Reference, Schema Access Strategy: Use 'Schema Name' Property) but sending the message I come across the error:PublishKafkaRecord_2_6[id=XXX] Failed to send FlowFile[filename=XXX] to Kafka: org.apache.nifi.processor.exception.ProcessException: Could not determine the Avro Schema to use for writing the content
- Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Cannot write Confluent Schema Registry Reference because the Schema Identifier is not known
I can confirm the configuration for the for the AvroRecordSetWriter, ConfluentSchemaRegistry controllers, and PublishKafkaRecord processor are all configured correctly, as I can send the Kafka message just fine using the default Publish Strategy (Use Content as Record Value). It only occurs using Use Wrapper, and the ConfluentSchemaRegistry.
A workaround that has worked was for using JsonRecordSetWriter w/ embedded JSON schemas, but it would be nice to continue using our Avro Schema Registry for this.
I'd appreciate if someone had any advice or experience with this issue, otherwise I'd like to log an issue in JIRA.
Thank you,Austin Tao
Re: ASF Jira account creation, PublishKafkaRecord issue
Posted by Paul Grey <gr...@gmail.com>.
Hi. I worked that feature; I'll have a look.
When the "Publish Strategy" is set to "Use Wrapper", my recollection is
that a second record schema is needed for the key, depending on the
processor configuration as a whole. Are you able to share that processor
configuration?
Thanks
On Fri, Dec 16, 2022 at 3:46 PM Austin Tao via users <us...@nifi.apache.org>
wrote:
> To whom it may concern,
>
> Hello, I would like to report an issue for Nifi. But, following the new
> Jira Guidelines, I would therefore like to request that an account for ASF
> Jira in order to create a ticket.
>
> In regards to the bug, using Nifi 1.19.1 I would like to send a tombstone
> message (null payload) to Kafka and using the Confluent JDBC sink
> connector
> <https://docs.confluent.io/kafka-connectors/jdbc/current/sink-connector/sink_config_options.html> (with
> delete.enabled=true) to delete a record in our Postgres database. I believe
> as of Nifi 1.19, PublishKafkaRecord_2_6 now supports the '*Publish
> Strategy*: Use Wrapper' feature which allows setting the Kafka message
> key and value (Primary Key as the Kafka key, null for the Kafka value). For
> the Record Key Writer, I'm using an AvroRecordSetWriter to validate and
> serialize the key against the confluent schema registry (*Schema Write
> Strategy*: Confluent Schema Registry Reference, *Schema Access Strategy*:
> Use 'Schema Name' Property) but sending the message I come across the error:
> *PublishKafkaRecord_2_6[id=XXX] Failed to send FlowFile[filename=XXX] to
> Kafka: org.apache.nifi.processor.exception.ProcessException: Could not
> determine the Avro Schema to use for writing the content*
>
> *- Caused by: org.apache.nifi.schema.access.SchemaNotFoundException:
> Cannot write Confluent Schema Registry Reference because the Schema
> Identifier is not known*
>
>
> I can confirm the configuration for the for the AvroRecordSetWriter,
> ConfluentSchemaRegistry controllers, and PublishKafkaRecord processor are
> all configured correctly, as I can send the Kafka message just fine using
> the default *Publish Strategy* (Use Content as Record Value). It only
> occurs using Use Wrapper, and the ConfluentSchemaRegistry.
>
> A workaround that has worked was for using JsonRecordSetWriter w/ embedded
> JSON schemas, but it would be nice to continue using our Avro Schema
> Registry for this.
>
> I'd appreciate if someone had any advice or experience with this issue,
> otherwise I'd like to log an issue in JIRA.
>
> Thank you,
> Austin Tao
>
>