You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "hpvd (via GitHub)" <gi...@apache.org> on 2025/11/24 04:57:47 UTC
[D] Full automatic dynamic lakehouse sink [fluss]
GitHub user hpvd created a discussion: Full automatic dynamic lakehouse sink
Flink just released a full dynamic sink for iceberg which makes writing Kafka to iceberg at scale really easy:
- schema evolution
- new topic recovery
- routing multiple types from one topic
...
see https://flink.apache.org/2025/11/11/from-stream-to-lakehouse-kafka-ingestion-with-the-flink-dynamic-iceberg-sink/
Are there plans to make something like this possible in combination with all the Fluss advantages?
Looks very adventurous to me especially in combination with lance format, which easily allow evolution without data rewrite, fast point query and ai data of any modality..
GitHub link: https://github.com/apache/fluss/discussions/2011
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Full automatic dynamic lakehouse sink [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
GitHub user polyzos added a comment to the discussion: Full automatic dynamic lakehouse sink
@MehulBatra @luoyuxia, maybe you can help with this?
GitHub link: https://github.com/apache/fluss/discussions/2011#discussioncomment-15072289
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Full automatic dynamic lakehouse sink [fluss]
Posted by "hpvd (via GitHub)" <gi...@apache.org>.
GitHub user hpvd edited a discussion: Full automatic dynamic lakehouse sink
Flink just released a full dynamic sink for iceberg which makes writing Kafka to iceberg at scale really easy:
- schema evolution
- new topic recovery
- routing multiple types from one topic
...
see https://flink.apache.org/2025/11/11/from-stream-to-lakehouse-kafka-ingestion-with-the-flink-dynamic-iceberg-sink/
Are there plans to make something like this possible in combination with all the Fluss advantages?
Looks very adventurous to me especially in combination with [lance format](https://github.com/lance-format/lance) / lancedb, which easily allow evolution without data rewrite, fast point query and dealing with AI data of any modality...
GitHub link: https://github.com/apache/fluss/discussions/2011
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Full automatic dynamic lakehouse sink [fluss]
Posted by "wuchong (via GitHub)" <gi...@apache.org>.
GitHub user wuchong added a comment to the discussion: Full automatic dynamic lakehouse sink
Hi @hpvd,
The current lakehouse integration in Fluss is indeed **"dynamic"**, as you expected:
- **Automated schema evolution**: Any schema changes to a Fluss table are propagated to the corresponding Iceberg table in real time.
- **Automated new table tiering**: Any new Fluss table with `table.datalake.enabled=true` is automatically tiered to an Iceberg table by the Tiering Service (you don't need to start new jobs for the Tiering Service).
- **Routing multiple types from a single topic**: This is **not supported** in Fluss, as Fluss follows a **schema-centric table model**, each table has a well-defined schema, and Fluss does not mix records of different tables within one table.
GitHub link: https://github.com/apache/fluss/discussions/2011#discussioncomment-15465878
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Full automatic dynamic lakehouse sink [fluss]
Posted by "hpvd (via GitHub)" <gi...@apache.org>.
GitHub user hpvd closed a discussion: Full automatic dynamic lakehouse sink
Flink just released a full dynamic sink for iceberg which makes writing Kafka to iceberg at scale really easy:
- automated schema evolution
- automated new topic recovery
- routing multiple types from one topic
...
see https://flink.apache.org/2025/11/11/from-stream-to-lakehouse-kafka-ingestion-with-the-flink-dynamic-iceberg-sink/
Are there plans to make something like this possible in combination with all the Fluss advantages?
Looks very adventurous to me especially in combination with [lance format](https://github.com/lance-format/lance) / lancedb, which easily allow evolution without data rewrite, fast point query and dealing with AI data of any modality...
GitHub link: https://github.com/apache/fluss/discussions/2011
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Full automatic dynamic lakehouse sink [fluss]
Posted by "hpvd (via GitHub)" <gi...@apache.org>.
GitHub user hpvd edited a discussion: Full automatic dynamic lakehouse sink
Flink just released a full dynamic sink for iceberg which makes writing Kafka to iceberg at scale really easy:
- schema evolution
- new topic recovery
- routing multiple types from one topic
...
see https://flink.apache.org/2025/11/11/from-stream-to-lakehouse-kafka-ingestion-with-the-flink-dynamic-iceberg-sink/
Are there plans to make something like this possible in combination with all the Fluss advantages?
Looks very adventurous to me especially in combination with lance format, which easily allow evolution without data rewrite, fast point query and dealing with AI data of any modality...
GitHub link: https://github.com/apache/fluss/discussions/2011
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org
Re: [D] Full automatic dynamic lakehouse sink [fluss]
Posted by "hpvd (via GitHub)" <gi...@apache.org>.
GitHub user hpvd edited a discussion: Full automatic dynamic lakehouse sink
Flink just released a full dynamic sink for iceberg which makes writing Kafka to iceberg at scale really easy:
- automated schema evolution
- automated new topic recovery
- routing multiple types from one topic
...
see https://flink.apache.org/2025/11/11/from-stream-to-lakehouse-kafka-ingestion-with-the-flink-dynamic-iceberg-sink/
Are there plans to make something like this possible in combination with all the Fluss advantages?
Looks very adventurous to me especially in combination with [lance format](https://github.com/lance-format/lance) / lancedb, which easily allow evolution without data rewrite, fast point query and dealing with AI data of any modality...
GitHub link: https://github.com/apache/fluss/discussions/2011
----
This is an automatically sent email for issues@fluss.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@fluss.apache.org