You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Utkarsh Parekh <Ut...@albertsons.com> on 2025/02/26 07:51:15 UTC
Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Hi Jan,
The pipeline runs fine with viaSDF (the default connector), but I’m looking to switch to the legacy connector. The reason for this change is that with the SDF connector, the Dataflow job generates too many Kafka connections.
I’m referring to the documentation: https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html
[cid:image001.png@01DB87DF.AE556390]
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com
From: Jan Lukavský <je...@seznam.cz>
Date: Tuesday, February 25, 2025 at 11:37 PM
To: user@beam.apache.org <us...@beam.apache.org>
Subject: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Hi Utharsh,
the cause of the error is the use_deprecated_read flag you pass to the expansion service. Dataflow v2 expands KafkaIO using SDF [1] and does not support (as the error says) the depreacted Read transform. Just remove the flag and it should work.
Best,
Jan
[1] https://beam.apache.org/blog/splittable-do-fn/<https://urldefense.com/v3/__https:/beam.apache.org/blog/splittable-do-fn/__;!!K1aXqncla1X7G90AkdLmCg!tZNJz5Y8FJyXIB68tfMvEGmuqDTHZ7lrH51iHzSaNspZP-Ckez4a1IBDn-KMnGiTjA9z6iZBBGPY3zE0h9_4Nqs$>
On 2/26/25 08:31, Utkarsh Parekh wrote:
Hello,
I’m encountering the following error while working with ReadFromKafka with the Dataflow runner V2 in Apache Beam:
2025-02-25 23:14:19.435 PST
Error translating pipeline. Unknown non-composite transform urn beam:transform:read:v1 for external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
Below is the code snippet triggering the error:
from apache_beam.io.kafka import ReadFromKafka, default_io_expansion_service
pcoll = self.pipeline | "Reading messages from Kafka" >> (ReadFromKafka(
self.consumer_config,
self.kafka_config.get('topics'),
commit_offset_in_finalize=True,
with_metadata=True,
expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
)
Background:
I’m looking to switch from the default connector (ReadFromKafkaViaSDF) to the legacy connector (ReadFromKafkaViaUnboundedSource), as the default connector is causing the Dataflow job to generate many Kafka connections.
Any guidance or suggestions to resolve this issue would be greatly appreciated.
Thank you
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Posted by Utkarsh Parekh <Ut...@albertsons.com>.
Hi Jan,
I couldn’t find any logs like with the messages like - "Member ... sending LeaveGroup request to coordinator ... due to the consumer is being closed"?
Here’s what I’m seeing:
1. There are a lot of logs at step id - Reading messages from Kafka/KafkaIO.Read/KafkaIO.Read.ReadFromKafkaViaUnbounded/Read(KafkaUnboundedSource)/ParDo(UnboundedSourceAsSDFWrapper)/ParMultiDo(UnboundedSourceAsSDFWrapper)/ProcessElementAndRestrictionWithSizing
Example log entry:
Request joining group due to: consumer pro-actively leaving the group
1. At the step id - Reading messages from
Kafka/KafkaIO.Read/KafkaIO.Read.ReadFromKafkaViaUnbounded/Read(KafkaUnboundedSource)/ParDo(UnboundedSourceAsSDFWrapper)/ParMultiDo(UnboundedSourceAsSDFWrapper)/SplitWithSizing
These appear after the initialization step, occuring during the partition-based split process.
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com
From: Jan Lukavský <je...@seznam.cz>
Date: Wednesday, February 26, 2025 at 3:45 AM
To: user@beam.apache.org <us...@beam.apache.org>
Subject: Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Hi Utkarsh,
I think this is not expected. Readers should be cached in the restriction tracker [1], thus should not be closed and reopened when splitting each restriction. Do you see logs like these when the consumer leaves the group?
"Member ... sending LeaveGroup request to coordinator ... due to the consumer is being closed"?
if yes, you might try to investigate the logs there, it might help find the cause.
Jan
[1] https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L856<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java*L856__;Iw!!K1aXqncla1X7G90AkdLmCg!scKdEfoy816j9eR9mtj9G6TinnNzbFimS9LmsJq9wrbpXfJlrqS27V0gq_nZ46KV7SO0enKex1hx3NtFg5qKA-Y$>
On 2/26/25 11:16, Utkarsh Parekh wrote:
Hi Jan –
I checked the logs after an hour and observed the same behavior as with viaSDF. The consumer is proactively leaving the group, and this occurs 5,127 times. Each occurrence results in the creation of a new connection to Kafka.
Does this sound right?
[cid:part1.C8OIGwpS.n5y1hlvW@seznam.cz]
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
From: Jan Lukavský <je...@seznam.cz>
Date: Wednesday, February 26, 2025 at 1:31 AM
To: user@beam.apache.org<ma...@beam.apache.org> <us...@beam.apache.org>
Subject: Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
On 2/26/25 10:08, Utkarsh Parekh wrote:
Hi Jan,
It looks like the solution is working. Based on the Dataflow logs, I can see it’s creating only 7 connections, and the number remains stable without increasing.
However, when using viaSDF, the connections exceed 2000 and continue to grow.
Regarding the legacy mode, is my understanding correct that it first retrieves the number of partitions and then passes them to the DoFn to consumer messages from the Kafka topic?
Yes, that is the case for legacy Read transform, because it creates splits only during initialization.
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
From: Jan Lukavský <je...@seznam.cz>
Date: Wednesday, February 26, 2025 at 12:25 AM
To: user@beam.apache.org<ma...@beam.apache.org> <us...@beam.apache.org>
Subject: Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Apparently, Dataflow V2 does not support the Read transform, but there is a way around this. The Read transform has can be expanded to SDF using UnboundedSourceAsSDF, but the use_deprecated_read flag prevents this. To use legacy Read wrapped as SDF (which is supported on dataflow) use use_unbounded_sdf_wrapper flag instead. Seems there is some room for improvement of documentation, because this is currectly documented only in code of the KafkaIO [1].
Jan
[1] https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1572<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java*L1572__;Iw!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11wSROCRTs$>
On 2/26/25 08:51, Utkarsh Parekh wrote:
Hi Jan,
The pipeline runs fine with viaSDF (the default connector), but I’m looking to switch to the legacy connector. The reason for this change is that with the SDF connector, the Dataflow job generates too many Kafka connections.
I’m referring to the documentation: https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html<https://urldefense.com/v3/__https:/beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html__;!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11w53QxC10$>
[cid:part2.Rj1ceQq9.LEVrmqlr@seznam.cz]
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
From: Jan Lukavský <je...@seznam.cz>
Date: Tuesday, February 25, 2025 at 11:37 PM
To: user@beam.apache.org<ma...@beam.apache.org> <us...@beam.apache.org>
Subject: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Hi Utharsh,
the cause of the error is the use_deprecated_read flag you pass to the expansion service. Dataflow v2 expands KafkaIO using SDF [1] and does not support (as the error says) the depreacted Read transform. Just remove the flag and it should work.
Best,
Jan
[1] https://beam.apache.org/blog/splittable-do-fn/<https://urldefense.com/v3/__https:/beam.apache.org/blog/splittable-do-fn/__;!!K1aXqncla1X7G90AkdLmCg!tZNJz5Y8FJyXIB68tfMvEGmuqDTHZ7lrH51iHzSaNspZP-Ckez4a1IBDn-KMnGiTjA9z6iZBBGPY3zE0h9_4Nqs$>
On 2/26/25 08:31, Utkarsh Parekh wrote:
Hello,
I’m encountering the following error while working with ReadFromKafka with the Dataflow runner V2 in Apache Beam:
2025-02-25 23:14:19.435 PST
Error translating pipeline. Unknown non-composite transform urn beam:transform:read:v1 for external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
Below is the code snippet triggering the error:
from apache_beam.io.kafka import ReadFromKafka, default_io_expansion_service
pcoll = self.pipeline | "Reading messages from Kafka" >> (ReadFromKafka(
self.consumer_config,
self.kafka_config.get('topics'),
commit_offset_in_finalize=True,
with_metadata=True,
expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
)
Background:
I’m looking to switch from the default connector (ReadFromKafkaViaSDF) to the legacy connector (ReadFromKafkaViaUnboundedSource), as the default connector is causing the Dataflow job to generate many Kafka connections.
Any guidance or suggestions to resolve this issue would be greatly appreciated.
Thank you
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Posted by Jan Lukavský <je...@seznam.cz>.
Hi Utkarsh,
I think this is not expected. Readers should be cached in the
restriction tracker [1], thus should not be closed and reopened when
splitting each restriction. Do you see logs like these when the consumer
leaves the group?
"Member ... sending LeaveGroup request to coordinator ... due to the
consumer is being closed"?
if yes, you might try to investigate the logs there, it might help find
the cause.
Jan
[1]
https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L856
On 2/26/25 11:16, Utkarsh Parekh wrote:
>
> Hi Jan –
>
> I checked the logs after an hour and observed the same behavior as
> with viaSDF. The consumer is proactively leaving the group, and this
> occurs 5,127 times. Each occurrence results in the creation of a new
> connection to Kafka.
>
> Does this sound right?
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> *From: *Jan Lukavský <je...@seznam.cz>
> *Date: *Wednesday, February 26, 2025 at 1:31 AM
> *To: *user@beam.apache.org <us...@beam.apache.org>
> *Subject: *Re: EXTERNAL Email: Re: Assistance required: Error
> Translating Pipeline with ReadFromKafka in Apache Beam
>
> On 2/26/25 10:08, Utkarsh Parekh wrote:
>
> Hi Jan,
>
> It looks like the solution is working. Based on the Dataflow
> logs, I can see it’s creating only 7 connections, and the number
> remains stable without increasing.
>
> However, when using viaSDF, the connections exceed 2000 and
> continue to grow.
>
> Regarding the legacy mode, is my understanding correct that it
> first retrieves the number of partitions and then passes them to
> the DoFn to consumer messages from the Kafka topic?
>
> Yes, that is the case for legacy Read transform, because it creates
> splits only during initialization.
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> *From: *Jan Lukavský <je...@seznam.cz> <ma...@seznam.cz>
> *Date: *Wednesday, February 26, 2025 at 12:25 AM
> *To: *user@beam.apache.org <us...@beam.apache.org>
> <ma...@beam.apache.org>
> *Subject: *Re: EXTERNAL Email: Re: Assistance required: Error
> Translating Pipeline with ReadFromKafka in Apache Beam
>
> Apparently, Dataflow V2 does not support the Read transform, but
> there is a way around this. The Read transform has can be expanded
> to SDF using UnboundedSourceAsSDF, but the use_deprecated_read
> flag prevents this. To use legacy Read wrapped as SDF (which is
> supported on dataflow) use use_unbounded_sdf_wrapper flag instead.
> Seems there is some room for improvement of documentation, because
> this is currectly documented only in code of the KafkaIO [1].
>
> Jan
>
> [1]
> https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1572
> <https://urldefense.com/v3/__https:/github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java*L1572__;Iw!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11wSROCRTs$>
>
> On 2/26/25 08:51, Utkarsh Parekh wrote:
>
> Hi Jan,
>
> The pipeline runs fine with viaSDF (the default connector),
> but I’m looking to switch to the legacy connector. The reason
> for this change is that with the SDF connector, the Dataflow
> job generates too many Kafka connections.
>
> I’m referring to the documentation:
> https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html
> <https://urldefense.com/v3/__https:/beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html__;!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11w53QxC10$>
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> *From: *Jan Lukavský <je...@seznam.cz> <ma...@seznam.cz>
> *Date: *Tuesday, February 25, 2025 at 11:37 PM
> *To: *user@beam.apache.org <us...@beam.apache.org>
> <ma...@beam.apache.org>
> *Subject: *EXTERNAL Email: Re: Assistance required: Error
> Translating Pipeline with ReadFromKafka in Apache Beam
>
> Hi Utharsh,
>
> the cause of the error is the use_deprecated_read flag you
> pass to the expansion service. Dataflow v2 expands KafkaIO
> using SDF [1] and does not support (as the error says) the
> depreacted Read transform. Just remove the flag and it should
> work.
>
> Best,
>
> Jan
>
> [1] https://beam.apache.org/blog/splittable-do-fn/
> <https://urldefense.com/v3/__https:/beam.apache.org/blog/splittable-do-fn/__;!!K1aXqncla1X7G90AkdLmCg!tZNJz5Y8FJyXIB68tfMvEGmuqDTHZ7lrH51iHzSaNspZP-Ckez4a1IBDn-KMnGiTjA9z6iZBBGPY3zE0h9_4Nqs$>
>
> On 2/26/25 08:31, Utkarsh Parekh wrote:
>
> Hello,
>
> I’m encountering the following error while working with
> ReadFromKafka with the Dataflow runner V2 in Apache Beam:
>
> 2025-02-25 23:14:19.435 PST
>
> Error translating pipeline. Unknown non-composite
> transform urn beam:transform:read:v1for
> external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
>
> Below is the code snippet triggering the error:
>
> fromapache_beam.io.kafka importReadFromKafka,
> default_io_expansion_service
>
> pcoll =self.pipeline |"Reading messages from
> Kafka">>(ReadFromKafka(
>
> self.consumer_config,
>
> self.kafka_config.get('topics'),
>
> commit_offset_in_finalize=True,
>
> with_metadata=True,
>
> expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
>
> )
>
> *Background:*
>
> I’m looking to switch from the default connector
> (ReadFromKafkaViaSDF) to the legacy connector
> (ReadFromKafkaViaUnboundedSource), as the default
> connector is causing the Dataflow job to generate many
> Kafka connections.
>
> Any guidance or suggestions to resolve this issue would be
> greatly appreciated.
>
> Thank you
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> ------------------------------------------------------------------------
>
> Warning: All e-mail sent to this address will be received
> by the corporate e-mail system, and is subject to archival
> and review by someone other than the recipient. This
> e-mail may contain proprietary information and is intended
> only for the use of the intended recipient(s). If the
> reader of this message is not the intended recipient(s),
> you are notified that you have received this message in
> error and that any review, dissemination, distribution or
> copying of this message is strictly prohibited. If you
> have received this message in error, please notify the
> sender immediately.
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
>
> Warning: All e-mail sent to this address will be received by
> the corporate e-mail system, and is subject to archival and
> review by someone other than the recipient. This e-mail may
> contain proprietary information and is intended only for the
> use of the intended recipient(s). If the reader of this
> message is not the intended recipient(s), you are notified
> that you have received this message in error and that any
> review, dissemination, distribution or copying of this message
> is strictly prohibited. If you have received this message in
> error, please notify the sender immediately.
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
>
> Warning: All e-mail sent to this address will be received by the
> corporate e-mail system, and is subject to archival and review by
> someone other than the recipient. This e-mail may contain
> proprietary information and is intended only for the use of the
> intended recipient(s). If the reader of this message is not the
> intended recipient(s), you are notified that you have received
> this message in error and that any review, dissemination,
> distribution or copying of this message is strictly prohibited. If
> you have received this message in error, please notify the sender
> immediately.
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
> Warning: All e-mail sent to this address will be received by the
> corporate e-mail system, and is subject to archival and review by
> someone other than the recipient. This e-mail may contain proprietary
> information and is intended only for the use of the intended
> recipient(s). If the reader of this message is not the intended
> recipient(s), you are notified that you have received this message in
> error and that any review, dissemination, distribution or copying of
> this message is strictly prohibited. If you have received this message
> in error, please notify the sender immediately.
> ------------------------------------------------------------------------
Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Posted by Utkarsh Parekh <Ut...@albertsons.com>.
Hi Jan –
I checked the logs after an hour and observed the same behavior as with viaSDF. The consumer is proactively leaving the group, and this occurs 5,127 times. Each occurrence results in the creation of a new connection to Kafka.
Does this sound right?
[cid:image001.png@01DB87F3.E97BA9C0]
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com
From: Jan Lukavský <je...@seznam.cz>
Date: Wednesday, February 26, 2025 at 1:31 AM
To: user@beam.apache.org <us...@beam.apache.org>
Subject: Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
On 2/26/25 10:08, Utkarsh Parekh wrote:
Hi Jan,
It looks like the solution is working. Based on the Dataflow logs, I can see it’s creating only 7 connections, and the number remains stable without increasing.
However, when using viaSDF, the connections exceed 2000 and continue to grow.
Regarding the legacy mode, is my understanding correct that it first retrieves the number of partitions and then passes them to the DoFn to consumer messages from the Kafka topic?
Yes, that is the case for legacy Read transform, because it creates splits only during initialization.
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
From: Jan Lukavský <je...@seznam.cz>
Date: Wednesday, February 26, 2025 at 12:25 AM
To: user@beam.apache.org<ma...@beam.apache.org> <us...@beam.apache.org>
Subject: Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Apparently, Dataflow V2 does not support the Read transform, but there is a way around this. The Read transform has can be expanded to SDF using UnboundedSourceAsSDF, but the use_deprecated_read flag prevents this. To use legacy Read wrapped as SDF (which is supported on dataflow) use use_unbounded_sdf_wrapper flag instead. Seems there is some room for improvement of documentation, because this is currectly documented only in code of the KafkaIO [1].
Jan
[1] https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1572<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java*L1572__;Iw!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11wSROCRTs$>
On 2/26/25 08:51, Utkarsh Parekh wrote:
Hi Jan,
The pipeline runs fine with viaSDF (the default connector), but I’m looking to switch to the legacy connector. The reason for this change is that with the SDF connector, the Dataflow job generates too many Kafka connections.
I’m referring to the documentation: https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html<https://urldefense.com/v3/__https:/beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html__;!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11w53QxC10$>
[cid:part1.btYAUlJB.KvdtL6JY@seznam.cz]
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
From: Jan Lukavský <je...@seznam.cz>
Date: Tuesday, February 25, 2025 at 11:37 PM
To: user@beam.apache.org<ma...@beam.apache.org> <us...@beam.apache.org>
Subject: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Hi Utharsh,
the cause of the error is the use_deprecated_read flag you pass to the expansion service. Dataflow v2 expands KafkaIO using SDF [1] and does not support (as the error says) the depreacted Read transform. Just remove the flag and it should work.
Best,
Jan
[1] https://beam.apache.org/blog/splittable-do-fn/<https://urldefense.com/v3/__https:/beam.apache.org/blog/splittable-do-fn/__;!!K1aXqncla1X7G90AkdLmCg!tZNJz5Y8FJyXIB68tfMvEGmuqDTHZ7lrH51iHzSaNspZP-Ckez4a1IBDn-KMnGiTjA9z6iZBBGPY3zE0h9_4Nqs$>
On 2/26/25 08:31, Utkarsh Parekh wrote:
Hello,
I’m encountering the following error while working with ReadFromKafka with the Dataflow runner V2 in Apache Beam:
2025-02-25 23:14:19.435 PST
Error translating pipeline. Unknown non-composite transform urn beam:transform:read:v1 for external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
Below is the code snippet triggering the error:
from apache_beam.io.kafka import ReadFromKafka, default_io_expansion_service
pcoll = self.pipeline | "Reading messages from Kafka" >> (ReadFromKafka(
self.consumer_config,
self.kafka_config.get('topics'),
commit_offset_in_finalize=True,
with_metadata=True,
expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
)
Background:
I’m looking to switch from the default connector (ReadFromKafkaViaSDF) to the legacy connector (ReadFromKafkaViaUnboundedSource), as the default connector is causing the Dataflow job to generate many Kafka connections.
Any guidance or suggestions to resolve this issue would be greatly appreciated.
Thank you
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Posted by Jan Lukavský <je...@seznam.cz>.
On 2/26/25 10:08, Utkarsh Parekh wrote:
>
> Hi Jan,
>
> It looks like the solution is working. Based on the Dataflow logs, I
> can see it’s creating only 7 connections, and the number remains
> stable without increasing.
>
> However, when using viaSDF, the connections exceed 2000 and continue
> to grow.
>
> Regarding the legacy mode, is my understanding correct that it first
> retrieves the number of partitions and then passes them to the DoFn to
> consumer messages from the Kafka topic?
>
Yes, that is the case for legacy Read transform, because it creates
splits only during initialization.
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> *From: *Jan Lukavský <je...@seznam.cz>
> *Date: *Wednesday, February 26, 2025 at 12:25 AM
> *To: *user@beam.apache.org <us...@beam.apache.org>
> *Subject: *Re: EXTERNAL Email: Re: Assistance required: Error
> Translating Pipeline with ReadFromKafka in Apache Beam
>
> Apparently, Dataflow V2 does not support the Read transform, but there
> is a way around this. The Read transform has can be expanded to SDF
> using UnboundedSourceAsSDF, but the use_deprecated_read flag prevents
> this. To use legacy Read wrapped as SDF (which is supported on
> dataflow) use use_unbounded_sdf_wrapper flag instead. Seems there is
> some room for improvement of documentation, because this is currectly
> documented only in code of the KafkaIO [1].
>
> Jan
>
> [1]
> https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1572
> <https://urldefense.com/v3/__https:/github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java*L1572__;Iw!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11wSROCRTs$>
>
> On 2/26/25 08:51, Utkarsh Parekh wrote:
>
> Hi Jan,
>
> The pipeline runs fine with viaSDF (the default connector), but
> I’m looking to switch to the legacy connector. The reason for this
> change is that with the SDF connector, the Dataflow job generates
> too many Kafka connections.
>
> I’m referring to the documentation:
> https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html
> <https://urldefense.com/v3/__https:/beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html__;!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11w53QxC10$>
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> *From: *Jan Lukavský <je...@seznam.cz> <ma...@seznam.cz>
> *Date: *Tuesday, February 25, 2025 at 11:37 PM
> *To: *user@beam.apache.org <us...@beam.apache.org>
> <ma...@beam.apache.org>
> *Subject: *EXTERNAL Email: Re: Assistance required: Error
> Translating Pipeline with ReadFromKafka in Apache Beam
>
> Hi Utharsh,
>
> the cause of the error is the use_deprecated_read flag you pass to
> the expansion service. Dataflow v2 expands KafkaIO using SDF [1]
> and does not support (as the error says) the depreacted Read
> transform. Just remove the flag and it should work.
>
> Best,
>
> Jan
>
> [1] https://beam.apache.org/blog/splittable-do-fn/
> <https://urldefense.com/v3/__https:/beam.apache.org/blog/splittable-do-fn/__;!!K1aXqncla1X7G90AkdLmCg!tZNJz5Y8FJyXIB68tfMvEGmuqDTHZ7lrH51iHzSaNspZP-Ckez4a1IBDn-KMnGiTjA9z6iZBBGPY3zE0h9_4Nqs$>
>
> On 2/26/25 08:31, Utkarsh Parekh wrote:
>
> Hello,
>
> I’m encountering the following error while working with
> ReadFromKafka with the Dataflow runner V2 in Apache Beam:
>
> 2025-02-25 23:14:19.435 PST
>
> Error translating pipeline. Unknown non-composite transform
> urn beam:transform:read:v1for
> external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
>
> Below is the code snippet triggering the error:
>
> fromapache_beam.io.kafka importReadFromKafka,
> default_io_expansion_service
>
> pcoll =self.pipeline |"Reading messages from
> Kafka">>(ReadFromKafka(
>
> self.consumer_config,
>
> self.kafka_config.get('topics'),
>
> commit_offset_in_finalize=True,
>
> with_metadata=True,
>
> expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
>
> )
>
> *Background:*
>
> I’m looking to switch from the default connector
> (ReadFromKafkaViaSDF) to the legacy connector
> (ReadFromKafkaViaUnboundedSource), as the default connector is
> causing the Dataflow job to generate many Kafka connections.
>
> Any guidance or suggestions to resolve this issue would be
> greatly appreciated.
>
> Thank you
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> ------------------------------------------------------------------------
>
> Warning: All e-mail sent to this address will be received by
> the corporate e-mail system, and is subject to archival and
> review by someone other than the recipient. This e-mail may
> contain proprietary information and is intended only for the
> use of the intended recipient(s). If the reader of this
> message is not the intended recipient(s), you are notified
> that you have received this message in error and that any
> review, dissemination, distribution or copying of this message
> is strictly prohibited. If you have received this message in
> error, please notify the sender immediately.
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
>
> Warning: All e-mail sent to this address will be received by the
> corporate e-mail system, and is subject to archival and review by
> someone other than the recipient. This e-mail may contain
> proprietary information and is intended only for the use of the
> intended recipient(s). If the reader of this message is not the
> intended recipient(s), you are notified that you have received
> this message in error and that any review, dissemination,
> distribution or copying of this message is strictly prohibited. If
> you have received this message in error, please notify the sender
> immediately.
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
> Warning: All e-mail sent to this address will be received by the
> corporate e-mail system, and is subject to archival and review by
> someone other than the recipient. This e-mail may contain proprietary
> information and is intended only for the use of the intended
> recipient(s). If the reader of this message is not the intended
> recipient(s), you are notified that you have received this message in
> error and that any review, dissemination, distribution or copying of
> this message is strictly prohibited. If you have received this message
> in error, please notify the sender immediately.
> ------------------------------------------------------------------------
Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Posted by Utkarsh Parekh <Ut...@albertsons.com>.
Hi Jan,
It looks like the solution is working. Based on the Dataflow logs, I can see it’s creating only 7 connections, and the number remains stable without increasing.
However, when using viaSDF, the connections exceed 2000 and continue to grow.
Regarding the legacy mode, is my understanding correct that it first retrieves the number of partitions and then passes them to the DoFn to consumer messages from the Kafka topic?
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com
From: Jan Lukavský <je...@seznam.cz>
Date: Wednesday, February 26, 2025 at 12:25 AM
To: user@beam.apache.org <us...@beam.apache.org>
Subject: Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Apparently, Dataflow V2 does not support the Read transform, but there is a way around this. The Read transform has can be expanded to SDF using UnboundedSourceAsSDF, but the use_deprecated_read flag prevents this. To use legacy Read wrapped as SDF (which is supported on dataflow) use use_unbounded_sdf_wrapper flag instead. Seems there is some room for improvement of documentation, because this is currectly documented only in code of the KafkaIO [1].
Jan
[1] https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1572<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java*L1572__;Iw!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11wSROCRTs$>
On 2/26/25 08:51, Utkarsh Parekh wrote:
Hi Jan,
The pipeline runs fine with viaSDF (the default connector), but I’m looking to switch to the legacy connector. The reason for this change is that with the SDF connector, the Dataflow job generates too many Kafka connections.
I’m referring to the documentation: https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html<https://urldefense.com/v3/__https:/beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html__;!!K1aXqncla1X7G90AkdLmCg!ohaayqnWxMZMhU-HczUWpfCUT8S_xZYiIIj2VsIaxKl_wVNvAT8zeVjQc3R75TxirG0opzJvSvfMF11w53QxC10$>
[cid:part1.HIO5b2BS.tkvIjo3X@seznam.cz]
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
From: Jan Lukavský <je...@seznam.cz>
Date: Tuesday, February 25, 2025 at 11:37 PM
To: user@beam.apache.org<ma...@beam.apache.org> <us...@beam.apache.org>
Subject: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Hi Utharsh,
the cause of the error is the use_deprecated_read flag you pass to the expansion service. Dataflow v2 expands KafkaIO using SDF [1] and does not support (as the error says) the depreacted Read transform. Just remove the flag and it should work.
Best,
Jan
[1] https://beam.apache.org/blog/splittable-do-fn/<https://urldefense.com/v3/__https:/beam.apache.org/blog/splittable-do-fn/__;!!K1aXqncla1X7G90AkdLmCg!tZNJz5Y8FJyXIB68tfMvEGmuqDTHZ7lrH51iHzSaNspZP-Ckez4a1IBDn-KMnGiTjA9z6iZBBGPY3zE0h9_4Nqs$>
On 2/26/25 08:31, Utkarsh Parekh wrote:
Hello,
I’m encountering the following error while working with ReadFromKafka with the Dataflow runner V2 in Apache Beam:
2025-02-25 23:14:19.435 PST
Error translating pipeline. Unknown non-composite transform urn beam:transform:read:v1 for external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
Below is the code snippet triggering the error:
from apache_beam.io.kafka import ReadFromKafka, default_io_expansion_service
pcoll = self.pipeline | "Reading messages from Kafka" >> (ReadFromKafka(
self.consumer_config,
self.kafka_config.get('topics'),
commit_offset_in_finalize=True,
with_metadata=True,
expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
)
Background:
I’m looking to switch from the default connector (ReadFromKafkaViaSDF) to the legacy connector (ReadFromKafkaViaUnboundedSource), as the default connector is causing the Dataflow job to generate many Kafka connections.
Any guidance or suggestions to resolve this issue would be greatly appreciated.
Thank you
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: Utkarsh.parekh@albertsons.com<ma...@albertsons.com>
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
________________________________
Warning: All e-mail sent to this address will be received by the corporate e-mail system, and is subject to archival and review by someone other than the recipient. This e-mail may contain proprietary information and is intended only for the use of the intended recipient(s). If the reader of this message is not the intended recipient(s), you are notified that you have received this message in error and that any review, dissemination, distribution or copying of this message is strictly prohibited. If you have received this message in error, please notify the sender immediately.
________________________________
Re: EXTERNAL Email: Re: Assistance required: Error Translating Pipeline with ReadFromKafka in Apache Beam
Posted by Jan Lukavský <je...@seznam.cz>.
Apparently, Dataflow V2 does not support the Read transform, but there
is a way around this. The Read transform has can be expanded to SDF
using UnboundedSourceAsSDF, but the use_deprecated_read flag prevents
this. To use legacy Read wrapped as SDF (which is supported on dataflow)
use use_unbounded_sdf_wrapper flag instead. Seems there is some room for
improvement of documentation, because this is currectly documented only
in code of the KafkaIO [1].
Jan
[1]
https://github.com/apache/beam/blob/7644a824da7c1b8c1293d8c61ead8fe4cd31e7ca/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L1572
On 2/26/25 08:51, Utkarsh Parekh wrote:
>
> Hi Jan,
>
> The pipeline runs fine with viaSDF (the default connector), but I’m
> looking to switch to the legacy connector. The reason for this change
> is that with the SDF connector, the Dataflow job generates too many
> Kafka connections.
>
> I’m referring to the documentation:
> https://beam.apache.org/releases/pydoc/current/apache_beam.io.kafka.html
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> *From: *Jan Lukavský <je...@seznam.cz>
> *Date: *Tuesday, February 25, 2025 at 11:37 PM
> *To: *user@beam.apache.org <us...@beam.apache.org>
> *Subject: *EXTERNAL Email: Re: Assistance required: Error Translating
> Pipeline with ReadFromKafka in Apache Beam
>
> Hi Utharsh,
>
> the cause of the error is the use_deprecated_read flag you pass to the
> expansion service. Dataflow v2 expands KafkaIO using SDF [1] and does
> not support (as the error says) the depreacted Read transform. Just
> remove the flag and it should work.
>
> Best,
>
> Jan
>
> [1] https://beam.apache.org/blog/splittable-do-fn/
> <https://urldefense.com/v3/__https:/beam.apache.org/blog/splittable-do-fn/__;!!K1aXqncla1X7G90AkdLmCg!tZNJz5Y8FJyXIB68tfMvEGmuqDTHZ7lrH51iHzSaNspZP-Ckez4a1IBDn-KMnGiTjA9z6iZBBGPY3zE0h9_4Nqs$>
>
> On 2/26/25 08:31, Utkarsh Parekh wrote:
>
> Hello,
>
> I’m encountering the following error while working with
> ReadFromKafka with the Dataflow runner V2 in Apache Beam:
>
> 2025-02-25 23:14:19.435 PST
>
> Error translating pipeline. Unknown non-composite transform urn
> beam:transform:read:v1 for
> external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
>
> Below is the code snippet triggering the error:
>
> fromapache_beam.io.kafka importReadFromKafka,
> default_io_expansion_service
>
> pcoll =self.pipeline |"Reading messages from
> Kafka">>(ReadFromKafka(
>
> self.consumer_config,
>
> self.kafka_config.get('topics'),
>
> commit_offset_in_finalize=True,
>
> with_metadata=True,
>
> expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
>
> )
>
> *Background:*
>
> I’m looking to switch from the default connector
> (ReadFromKafkaViaSDF) to the legacy connector
> (ReadFromKafkaViaUnboundedSource), as the default connector is
> causing the Dataflow job to generate many Kafka connections.
>
> Any guidance or suggestions to resolve this issue would be greatly
> appreciated.
>
> Thank you
>
> --
>
> Utkarsh
>
> Sr. Data Engineer
>
> Albertsons Companies
>
> Mobile: (860)-578-5208
>
> Email: Utkarsh.parekh@albertsons.com
>
> ------------------------------------------------------------------------
>
> Warning: All e-mail sent to this address will be received by the
> corporate e-mail system, and is subject to archival and review by
> someone other than the recipient. This e-mail may contain
> proprietary information and is intended only for the use of the
> intended recipient(s). If the reader of this message is not the
> intended recipient(s), you are notified that you have received
> this message in error and that any review, dissemination,
> distribution or copying of this message is strictly prohibited. If
> you have received this message in error, please notify the sender
> immediately.
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
> Warning: All e-mail sent to this address will be received by the
> corporate e-mail system, and is subject to archival and review by
> someone other than the recipient. This e-mail may contain proprietary
> information and is intended only for the use of the intended
> recipient(s). If the reader of this message is not the intended
> recipient(s), you are notified that you have received this message in
> error and that any review, dissemination, distribution or copying of
> this message is strictly prohibited. If you have received this message
> in error, please notify the sender immediately.
> ------------------------------------------------------------------------