[jira] [Updated] (FLINK-40108) Add Splitters for snapshot a table or database

["João Boto (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

João Boto updated FLINK-40108:
------------------------------
    Description: 
Historically, extracting large datasets or entire databases in Apache Flink required a fair amount of manual heavy lifting. Developers often had to write custom SQL queries and explicitly define chunking/partitioning strategies to prevent memory bottlenecks and ensure parallel execution.

With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this paradigm shifts. We now have a more native, automated way to handle full-table and full-database extractions seamlessly.
h3. The Old Way vs. The New Way
|*Feature*|*The Old Approach*|*With SplitterEnumerator*|
|*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for ranges.|Automated metadata-driven table/database scanning.|
|*Chunking Logic*|Custom-coded pagination or partitioning boundaries.|Built-in, dynamic split generation handled by the enumerator.|
|*Maintenance*|High. Schema changes or data volume spikes required manual tuning.|Low. Adapts dynamically to the underlying data structure.|
h3. Key Benefits
 * *Zero-Configuration Chunking:* You no longer need to guess the optimal chunk sizes or write complex boundary logic. The {{SplitterEnumerator}} automatically determines how to divide the table into manageable "splits" for parallel workers.

 * *Database-Wide Scaling:* Instead of writing dozens of individual source queries for a multi-table database, the enumerator can discover and orchestrate the extraction of all tables under the hood.

 * *Cleaner Codebases:* Removes hundreds of lines of boilerplate SQL and partitioning code, making your Flink jobs significantly easier to read, maintain, and audit.

h3. How it Works (Under the Hood)

Instead of the coordinator node just passing a static query to the source workers, the {{SplitterEnumerator}} acts as an intelligent traffic controller:
 # *Discovery:* It inspects the target database or table metadata.

 # *Splitting:* It breaks the dataset down into independent, parallelizable pieces (splits) based on primary keys, indices, or data volume.

 # *Distribution:* It dynamically assigns these splits to the source readers as they become available, ensuring an even workload distribution without manual intervention.

  was:
Historically, extracting large datasets or entire databases in Apache Flink required a fair amount of manual heavy lifting. Developers often had to write custom SQL queries and explicitly define chunking/partitioning strategies to prevent memory bottlenecks and ensure parallel execution.

With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this paradigm shifts. We now have a more native, automated way to handle full-table and full-database extractions seamlessly.
h3. The Old Way vs. The New Way
|*Feature*|*The Old Approach*|*With SplitterEnumerator*|
|*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for ranges.|Automated metadata-driven table/database scanning.|
|*Chunking Logic*|Custom-coded pagination or partitioning boundaries.|Built-in, dynamic split generation handled by the enumerator.|
|*Maintenance*|High. Schema changes or data volume spikes required manual tuning.|Low. Adapts dynamically to the underlying data structure.|


> Add Splitters for snapshot a table or database
> ----------------------------------------------
>
>                 Key: FLINK-40108
>                 URL: https://issues.apache.org/jira/browse/FLINK-40108
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / JDBC
>            Reporter: João Boto
>            Priority: Major
>
> Historically, extracting large datasets or entire databases in Apache Flink required a fair amount of manual heavy lifting. Developers often had to write custom SQL queries and explicitly define chunking/partitioning strategies to prevent memory bottlenecks and ensure parallel execution.
> With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this paradigm shifts. We now have a more native, automated way to handle full-table and full-database extractions seamlessly.
> h3. The Old Way vs. The New Way
> |*Feature*|*The Old Approach*|*With SplitterEnumerator*|
> |*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for ranges.|Automated metadata-driven table/database scanning.|
> |*Chunking Logic*|Custom-coded pagination or partitioning boundaries.|Built-in, dynamic split generation handled by the enumerator.|
> |*Maintenance*|High. Schema changes or data volume spikes required manual tuning.|Low. Adapts dynamically to the underlying data structure.|
> h3. Key Benefits
>  * *Zero-Configuration Chunking:* You no longer need to guess the optimal chunk sizes or write complex boundary logic. The {{SplitterEnumerator}} automatically determines how to divide the table into manageable "splits" for parallel workers.
>  * *Database-Wide Scaling:* Instead of writing dozens of individual source queries for a multi-table database, the enumerator can discover and orchestrate the extraction of all tables under the hood.
>  * *Cleaner Codebases:* Removes hundreds of lines of boilerplate SQL and partitioning code, making your Flink jobs significantly easier to read, maintain, and audit.
> h3. How it Works (Under the Hood)
> Instead of the coordinator node just passing a static query to the source workers, the {{SplitterEnumerator}} acts as an intelligent traffic controller:
>  # *Discovery:* It inspects the target database or table metadata.
>  # *Splitting:* It breaks the dataset down into independent, parallelizable pieces (splits) based on primary keys, indices, or data volume.
>  # *Distribution:* It dynamically assigns these splits to the source readers as they become available, ensuring an even workload distribution without manual intervention.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)