You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ling Li <ku...@gmail.com> on 2025/01/23 04:02:57 UTC
Urgent: Action Required for Iceberg Production
Hi team,
Happy New Year! I have a question regarding
the "org.apache.beam:beam-sdks-java-io-iceberg:2.61.0".
Does the org.apache.beam.sdk.schemas.Schema getDataSchema(String
destination) of class DynamicDestinations already exist?
Our team tried to use DynamicDestinations to decide which iceberg table to
write in the runtime but failed.
Attached is my code class DynamicIcebergDestinations. Our problem is that
we don't have a universal schema that can match all events to be
implemented for getDataSchema(). We need to use a parameter to get the
correct schema for each event, so we want to use: getDataSchema(String
destination). But seems getDataSchema(String destination) is not
implemented for org.apache.beam.sdk.io.iceberg.DynamicDestinations. Since
getDataSchema() will be automatically called and it cannot return a
universal schema, there is no way for us to
use IcebergIO.writeRows(icebergCatalogConfig).to(dynamicDestinations). I
also tried to use managed I/O connector, but I don't think
Managed.write support
with DynamicDestinations. Could you give some urgent guidance?
Thank you so much.
Sincerely,
Luyao
Re: Urgent: Action Required for Iceberg Production
Posted by Ling Li <ku...@gmail.com>.
Dear XQ and Ahmed,
We plan to use dynamic schemas and I tried to implement it by myself. But I
found maybe this is a blocker, Beam's Coder system generally requires a *fixed
schema* to serialize and deserialize data efficiently. When working with
dynamic schemas that are determined at runtime, this can become a blocker.
Is my understanding correct? Could you give some suggestions on how to
solve this?
Best regards,
Luyao
On Thu, Jan 23, 2025 at 10:01 AM XQ Hu <xq...@google.com> wrote:
> +kaikuaiqi@gmail.com explicitly. Please check the response.
>
> On Thu, Jan 23, 2025 at 10:45 AM Ahmed Abualsaud <
> ahmedabualsaud@apache.org> wrote:
>
>> Hey Luyao!
>>
>> Thanks for reaching out. Beam's Iceberg connector supports dynamic
>> destinations so long as they have equivalent Schemas. There's no support
>> for dynamic schemas yet, i.e. there's no `getDataSchema(<destination>)`
>> method.
>>
>> There's a feature request on Github:
>> https://github.com/apache/beam/issues/33724.
>>
>> > I don't think Managed.write support with DynamicDestinations
>>
>> ManagedIO does support dynamic destinations, in the format described
>> here:
>> https://cloud.google.com/dataflow/docs/guides/managed-io#dynamic-destinations.
>> It also does not support dynamic schemas yet.
>>
>> Best,
>> Ahmed
>>
>> On 2025/01/23 04:02:57 Ling Li wrote:
>> > Hi team,
>> >
>> > Happy New Year! I have a question regarding
>> > the "org.apache.beam:beam-sdks-java-io-iceberg:2.61.0".
>> >
>> > Does the org.apache.beam.sdk.schemas.Schema getDataSchema(String
>> > destination) of class DynamicDestinations already exist?
>> >
>> > Our team tried to use DynamicDestinations to decide which iceberg table
>> to
>> > write in the runtime but failed.
>> >
>> >
>> > Attached is my code class DynamicIcebergDestinations. Our problem is
>> that
>> > we don't have a universal schema that can match all events to be
>> > implemented for getDataSchema(). We need to use a parameter to get the
>> > correct schema for each event, so we want to use: getDataSchema(String
>> > destination). But seems getDataSchema(String destination) is not
>> > implemented for org.apache.beam.sdk.io.iceberg.DynamicDestinations.
>> Since
>> > getDataSchema() will be automatically called and it cannot return a
>> > universal schema, there is no way for us to
>> > use IcebergIO.writeRows(icebergCatalogConfig).to(dynamicDestinations). I
>> > also tried to use managed I/O connector, but I don't think
>> > Managed.write support
>> > with DynamicDestinations. Could you give some urgent guidance?
>> >
>> > Thank you so much.
>> >
>> > Sincerely,
>> > Luyao
>> >
>>
>
Re: Urgent: Action Required for Iceberg Production
Posted by XQ Hu via dev <de...@beam.apache.org>.
+kaikuaiqi@gmail.com explicitly. Please check the response.
On Thu, Jan 23, 2025 at 10:45 AM Ahmed Abualsaud <ah...@apache.org>
wrote:
> Hey Luyao!
>
> Thanks for reaching out. Beam's Iceberg connector supports dynamic
> destinations so long as they have equivalent Schemas. There's no support
> for dynamic schemas yet, i.e. there's no `getDataSchema(<destination>)`
> method.
>
> There's a feature request on Github:
> https://github.com/apache/beam/issues/33724.
>
> > I don't think Managed.write support with DynamicDestinations
>
> ManagedIO does support dynamic destinations, in the format described here:
> https://cloud.google.com/dataflow/docs/guides/managed-io#dynamic-destinations.
> It also does not support dynamic schemas yet.
>
> Best,
> Ahmed
>
> On 2025/01/23 04:02:57 Ling Li wrote:
> > Hi team,
> >
> > Happy New Year! I have a question regarding
> > the "org.apache.beam:beam-sdks-java-io-iceberg:2.61.0".
> >
> > Does the org.apache.beam.sdk.schemas.Schema getDataSchema(String
> > destination) of class DynamicDestinations already exist?
> >
> > Our team tried to use DynamicDestinations to decide which iceberg table
> to
> > write in the runtime but failed.
> >
> >
> > Attached is my code class DynamicIcebergDestinations. Our problem is that
> > we don't have a universal schema that can match all events to be
> > implemented for getDataSchema(). We need to use a parameter to get the
> > correct schema for each event, so we want to use: getDataSchema(String
> > destination). But seems getDataSchema(String destination) is not
> > implemented for org.apache.beam.sdk.io.iceberg.DynamicDestinations.
> Since
> > getDataSchema() will be automatically called and it cannot return a
> > universal schema, there is no way for us to
> > use IcebergIO.writeRows(icebergCatalogConfig).to(dynamicDestinations). I
> > also tried to use managed I/O connector, but I don't think
> > Managed.write support
> > with DynamicDestinations. Could you give some urgent guidance?
> >
> > Thank you so much.
> >
> > Sincerely,
> > Luyao
> >
>
Re: Urgent: Action Required for Iceberg Production
Posted by Ahmed Abualsaud <ah...@apache.org>.
Hey Luyao!
Thanks for reaching out. Beam's Iceberg connector supports dynamic destinations so long as they have equivalent Schemas. There's no support for dynamic schemas yet, i.e. there's no `getDataSchema(<destination>)` method.
There's a feature request on Github: https://github.com/apache/beam/issues/33724.
> I don't think Managed.write support with DynamicDestinations
ManagedIO does support dynamic destinations, in the format described here: https://cloud.google.com/dataflow/docs/guides/managed-io#dynamic-destinations. It also does not support dynamic schemas yet.
Best,
Ahmed
On 2025/01/23 04:02:57 Ling Li wrote:
> Hi team,
>
> Happy New Year! I have a question regarding
> the "org.apache.beam:beam-sdks-java-io-iceberg:2.61.0".
>
> Does the org.apache.beam.sdk.schemas.Schema getDataSchema(String
> destination) of class DynamicDestinations already exist?
>
> Our team tried to use DynamicDestinations to decide which iceberg table to
> write in the runtime but failed.
>
>
> Attached is my code class DynamicIcebergDestinations. Our problem is that
> we don't have a universal schema that can match all events to be
> implemented for getDataSchema(). We need to use a parameter to get the
> correct schema for each event, so we want to use: getDataSchema(String
> destination). But seems getDataSchema(String destination) is not
> implemented for org.apache.beam.sdk.io.iceberg.DynamicDestinations. Since
> getDataSchema() will be automatically called and it cannot return a
> universal schema, there is no way for us to
> use IcebergIO.writeRows(icebergCatalogConfig).to(dynamicDestinations). I
> also tried to use managed I/O connector, but I don't think
> Managed.write support
> with DynamicDestinations. Could you give some urgent guidance?
>
> Thank you so much.
>
> Sincerely,
> Luyao
>