You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Dave Hamilton <dh...@nanigans.com> on 2017/05/26 17:44:48 UTC

Kafka connector throughput reduction upon avro schema change

We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer throughput on the Kafka connector drop considerably. I am wondering if there is anything we can do to avoid such issues when we update schemas in the future?

This is what I believe is happening:


·         The avro producer application is running on 12 instances. They are restarted in a rolling fashion, switching from producing schema version 1 before the restart to schema version 2 afterward.

·         While the rolling restart is occurring, data on schema version 1 and schema version 2 is simultaneously being written to the topic.

·         The Kafka connector has to close the current avro file for a partition and ship it whenever it detects a schema change, which is happening several times due to the rolling nature of the schema update deployment and the mixture of message versions being written during this time. This process causes the overall consumer throughput to plummet.

Am I reasoning correctly about what we’re observing here? Is there any way to avoid this when we change schemas (short of stopping all instances of the service and bringing them up together on the new schema version)?

Thanks,
Dave

Re: Kafka connector throughput reduction upon avro schema change

Posted by Ewen Cheslack-Postava <ew...@confluent.io>.

What is your setting for schema.compatibility? I suspect the issue is
probably that it is defaulting to NONE which would cause the connector to
roll a new file when the schema changes (which will be frequent with data
that is interleaved with different schemas).

If you set it to BACKWARDS then the records would be properly projected and
not require rolling files. Of course this assumes you are ok with records
being projected to the latest schema.

-Ewen

On Thu, Jul 6, 2017 at 10:04 AM, Dave Hamilton <dh...@nanigans.com>
wrote:

> Bumping this. Has anyone here observed this in their Kafka connect
> deployments?
>
> Thanks,
> Dave
>
>
> On 5/26/17, 1:44 PM, "Dave Hamilton" <dh...@nanigans.com> wrote:
>
>     We are currently using the Kafka S3 connector to ship Avro data to S3.
> We made a change to one of our Avro schemas and have noticed consumer
> throughput on the Kafka connector drop considerably. I am wondering if
> there is anything we can do to avoid such issues when we update schemas in
> the future?
>
>     This is what I believe is happening:
>
>
>     ·         The avro producer application is running on 12 instances.
> They are restarted in a rolling fashion, switching from producing schema
> version 1 before the restart to schema version 2 afterward.
>
>     ·         While the rolling restart is occurring, data on schema
> version 1 and schema version 2 is simultaneously being written to the topic.
>
>     ·         The Kafka connector has to close the current avro file for a
> partition and ship it whenever it detects a schema change, which is
> happening several times due to the rolling nature of the schema update
> deployment and the mixture of message versions being written during this
> time. This process causes the overall consumer throughput to plummet.
>
>     Am I reasoning correctly about what we’re observing here? Is there any
> way to avoid this when we change schemas (short of stopping all instances
> of the service and bringing them up together on the new schema version)?
>
>     Thanks,
>     Dave
>
>
>
>

Re: Kafka connector throughput reduction upon avro schema change

Posted by Dave Hamilton <dh...@nanigans.com>.

Bumping this. Has anyone here observed this in their Kafka connect deployments?

Thanks,
Dave

On 5/26/17, 1:44 PM, "Dave Hamilton" <dh...@nanigans.com> wrote:

We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer throughput on the Kafka connector drop considerably. I am wondering if there is anything we can do to avoid such issues when we update schemas in the future?

This is what I believe is happening:

· The avro producer application is running on 12 instances. They are restarted in a rolling fashion, switching from producing schema version 1 before the restart to schema version 2 afterward.

· While the rolling restart is occurring, data on schema version 1 and schema version 2 is simultaneously being written to the topic.

· The Kafka connector has to close the current avro file for a partition and ship it whenever it detects a schema change, which is happening several times due to the rolling nature of the schema update deployment and the mixture of message versions being written during this time. This process causes the overall consumer throughput to plummet.

Am I reasoning correctly about what we’re observing here? Is there any way to avoid this when we change schemas (short of stopping all instances of the service and bringing them up together on the new schema version)?

Thanks,
Dave

Re: Kafka connector throughput reduction upon avro schema change

Posted by Dave Hamilton <dh...@nanigans.com>.

Hi, does anyone have advice on how to deal with this issue? Is it possible that changing a schema compatibility setting could correct it?

Thanks,
Dave

On 5/26/17, 1:44 PM, "Dave Hamilton" <dh...@nanigans.com> wrote:

This is what I believe is happening:

· The avro producer application is running on 12 instances. They are restarted in a rolling fashion, switching from producing schema version 1 before the restart to schema version 2 afterward.

· While the rolling restart is occurring, data on schema version 1 and schema version 2 is simultaneously being written to the topic.

Thanks,
Dave