You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Guitton Alexandre (Jira)" <ji...@apache.org> on 2020/08/05 08:55:00 UTC
[jira] [Updated] (NIFI-7707) [Processor/GCP - PubSub] Avro event
re-format when pushed to PubSub
[ https://issues.apache.org/jira/browse/NIFI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guitton Alexandre updated NIFI-7707:
------------------------------------
Description:
Hi,
as discussed on Slack : [https://apachenifi.slack.com/archives/C0L9VCD47/p1596463216211700]
When we generate an avro event from record and we publish it using _*PublishPubSub*_ processor the content-type of the file change from _*avro/binary*_ to *_UTF-8_* due to the following code part : [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/PublishGCPubSub.java#L157]
This make the event unreadable once Consumed :/
The workaround we found is base64 encoding the avro message, publishing it to PubSub, consuming it, and finally decoding it.
We expect this part of encoding would be handled by the processors.
was:
Hi,
as discussed on Slack : [https://apachenifi.slack.com/archives/C0L9VCD47/p1596463216211700]
When we generate an avro event from record and we publish it using _*PublishPubSub*_ processor the content-type of the file change from _*avro/binary*_ to *_UTF-8_* due to the following code part : [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/PublishGCPubSub.java#L157]
This make the event unreadable once Consumed :/
The workaround we found is base64 encoding the avro message, publishing it to PubSub, consuming it, and finally decoding it.
We expect this part of encoding would be handled by the processors.
> [Processor/GCP - PubSub] Avro event re-format when pushed to PubSub
> -------------------------------------------------------------------
>
> Key: NIFI-7707
> URL: https://issues.apache.org/jira/browse/NIFI-7707
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.11.4
> Reporter: Guitton Alexandre
> Priority: Major
> Labels: Avro, GCP
>
> Hi,
> as discussed on Slack : [https://apachenifi.slack.com/archives/C0L9VCD47/p1596463216211700]
> When we generate an avro event from record and we publish it using _*PublishPubSub*_ processor the content-type of the file change from _*avro/binary*_ to *_UTF-8_* due to the following code part : [https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/pubsub/PublishGCPubSub.java#L157]
> This make the event unreadable once Consumed :/
> The workaround we found is base64 encoding the avro message, publishing it to PubSub, consuming it, and finally decoding it.
> We expect this part of encoding would be handled by the processors.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)