You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Guosmilesmile (via GitHub)" <gi...@apache.org> on 2025/03/30 06:50:17 UTC
[PR] Doc: Add doc for flink exec config [iceberg]
Guosmilesmile opened a new pull request, #12691:
URL: https://github.com/apache/iceberg/pull/12691
We have observed that there are many parameters in FlinkConfigOptions, such as `table.exec.iceberg.infer-source-parallelism` and `table.exec.iceberg.use-flip27-source`, which are currently not documented on Flink’s configuration page. However, these configurations can significantly impact flink behaviors, such as the parallelism of Flink sources.
This PR aims to add these configurations to the Flink configuration documentation for better visibility and usability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#issuecomment-2860900277
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#issuecomment-2764417977
@pvary @stevenzwu @mxm Could you please take some time to review whether this change is feasible? Thanks！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026602432
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
+```
+
+| Flink configuration | Default | Description |
+|-------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If false, parallelism of source are set by config. If true, source parallelism is inferred according to splits number. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Sets max infer parallelism for source operator. |
+| table.exec.iceberg.expose-split-locality-info | null | If true, expose split host information to use Flink's locality aware split assigner. |
+| table.exec.iceberg.fetch-batch-record-count | 2048 | The target number of records for Iceberg reader fetch batch. |
+| table.exec.iceberg.worker-pool-size | ThreadPools.WORKER_THREAD_POOL_SIZE | The size of workers pool used to plan or scan manifests. If the value of Runtime.getRuntime().availableProcessors() is compared to 2, take the larger one. |
+| table.exec.iceberg.use-flip27-source | true | If true, Use the FLIP-27 based Iceberg source implementation. |
+| table.exec.iceberg.use-v2-sink | false | If true,Use the SinkV2 API based Iceberg sink implementation. |
+| table.exec.iceberg.split-assigner-type | SIMPLE | Split assigner type that determine how splits are assigned to readers. SIMPLE: simple assigner that doesn't provide any guarantee on order or locality. |
Review Comment:
I don't think we would like to expose this config to the users. It was a mistake, and we should aim for removing it in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026617326
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
+```
+
+| Flink configuration | Default | Description |
+|-------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If false, parallelism of source are set by config. If true, source parallelism is inferred according to splits number. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Sets max infer parallelism for source operator. |
Review Comment:
These are again source specific
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2020081147
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
Review Comment:
Even though these configurations start with table.exec, they can also directly be set by the Source and Sink in DataStream. That’s why I chose to use Exec options instead of Table options. If there’s a better suggestion, I think this can be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026603208
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
+```
+
+| Flink configuration | Default | Description |
+|-------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If false, parallelism of source are set by config. If true, source parallelism is inferred according to splits number. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Sets max infer parallelism for source operator. |
+| table.exec.iceberg.expose-split-locality-info | null | If true, expose split host information to use Flink's locality aware split assigner. |
+| table.exec.iceberg.fetch-batch-record-count | 2048 | The target number of records for Iceberg reader fetch batch. |
+| table.exec.iceberg.worker-pool-size | ThreadPools.WORKER_THREAD_POOL_SIZE | The size of workers pool used to plan or scan manifests. If the value of Runtime.getRuntime().availableProcessors() is compared to 2, take the larger one. |
+| table.exec.iceberg.use-flip27-source | true | If true, Use the FLIP-27 based Iceberg source implementation. |
Review Comment:
Source specific - maybe it should be at the source?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026600147
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
Review Comment:
This one is probably not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026616717
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
+```
+
+| Flink configuration | Default | Description |
+|-------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If false, parallelism of source are set by config. If true, source parallelism is inferred according to splits number. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Sets max infer parallelism for source operator. |
+| table.exec.iceberg.expose-split-locality-info | null | If true, expose split host information to use Flink's locality aware split assigner. |
Review Comment:
Is this working?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#issuecomment-2782366843
@Guosmilesmile: I did a little digging, and TBH, I don't understand why are these configs handled differently than the standard `FlinkWriteOptions` and `FlinkReadOptions`. Maybe we should ask @stevenzwu and @mxm and deprecated and migrate these configurations to there?
My current feeling is that we should migrate them to the standard read/write options, deprecate the old ones, and document them only in their final place. But I might miss some historical details 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026612267
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
+```
+
+| Flink configuration | Default | Description |
+|-------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If false, parallelism of source are set by config. If true, source parallelism is inferred according to splits number. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Sets max infer parallelism for source operator. |
+| table.exec.iceberg.expose-split-locality-info | null | If true, expose split host information to use Flink's locality aware split assigner. |
+| table.exec.iceberg.fetch-batch-record-count | 2048 | The target number of records for Iceberg reader fetch batch. |
+| table.exec.iceberg.worker-pool-size | ThreadPools.WORKER_THREAD_POOL_SIZE | The size of workers pool used to plan or scan manifests. If the value of Runtime.getRuntime().availableProcessors() is compared to 2, take the larger one. |
Review Comment:
The `Default` should be like this:
```
The number of the available processors, but minimally 2
```
And we can remove the `If` part of the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "Guosmilesmile (via GitHub)" <gi...@apache.org>.
Guosmilesmile commented on PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#issuecomment-2776038624
@pvary As you mentioned, some configs are about to expire, and some are mistakes. I’ve summarized all the configs:
**Source:**
table.exec.iceberg.infer-source-parallelism
table.exec.iceberg.infer-source-parallelism.max
table.exec.iceberg.fetch-batch-record-count
table.exec.iceberg.expose-split-locality-info
**Sink:**
table.exec.iceberg.use-v2-sink
**Source & Sink:**
table.exec.iceberg.worker-pool-size
**To drop:**
table.exec.iceberg.use-flip27-source
table.exec.iceberg.split-assigner-type
I think we don’t need a separate exec options. Instead, we can add the above configs directly to source options and sink options. The `table.exec.iceberg.worker-pool-size` is special and should appear in both source and sink options.
Do you think this is more suitable? Do you have any better suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026604541
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
+```
+
+| Flink configuration | Default | Description |
+|-------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If false, parallelism of source are set by config. If true, source parallelism is inferred according to splits number. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Sets max infer parallelism for source operator. |
+| table.exec.iceberg.expose-split-locality-info | null | If true, expose split host information to use Flink's locality aware split assigner. |
+| table.exec.iceberg.fetch-batch-record-count | 2048 | The target number of records for Iceberg reader fetch batch. |
+| table.exec.iceberg.worker-pool-size | ThreadPools.WORKER_THREAD_POOL_SIZE | The size of workers pool used to plan or scan manifests. If the value of Runtime.getRuntime().availableProcessors() is compared to 2, take the larger one. |
+| table.exec.iceberg.use-flip27-source | true | If true, Use the FLIP-27 based Iceberg source implementation. |
+| table.exec.iceberg.use-v2-sink | false | If true,Use the SinkV2 API based Iceberg sink implementation. |
Review Comment:
Sink specific. Maybe this should be at the sink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026594051
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
Review Comment:
I would defer this question for @stevenzwu and @mxm, as they manage many Flink users, but I would suggest to filter out the configurations which is really important for the users to avoid drowning them in information which is not useful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Doc: Add doc for flink exec config [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12691:
URL: https://github.com/apache/iceberg/pull/12691#discussion_r2026603875
##########
docs/docs/flink-configuration.md:
##########
@@ -198,4 +198,42 @@ they are.
This is only applicable to {@link StatisticsType#Map} for low-cardinality scenario. For
{@link StatisticsType#Sketch} high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
-write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
\ No newline at end of file
+write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Exec options
+
+When constructing Flink Iceberg source via Java API, configs can be set in Configuration like this:
+
+```
+configuration.setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+FlinkSource.forRowData()
+ .flinkConf(configuration)
+ ...
+```
+
+When using table API, options can be set in Flink's TableEnvironment.
+
+```
+TableEnvironment tEnv = createTableEnv();
+tEnv.getConfig()
+ .getConfiguration()
+ .setBoolean(FlinkConfigOptions.TABLE_EXEC_ICEBERG_INFER_SOURCE_PARALLELISM, true);
+```
+
+For Flink SQL, set options can be passed like this:
+```
+SET table.exec.iceberg.infer-source-parallelism.max=10;
+
+SELECT * FROM tableName;
+```
+
+| Flink configuration | Default | Description |
+|-------------------------------------------------|-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If false, parallelism of source are set by config. If true, source parallelism is inferred according to splits number. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Sets max infer parallelism for source operator. |
+| table.exec.iceberg.expose-split-locality-info | null | If true, expose split host information to use Flink's locality aware split assigner. |
+| table.exec.iceberg.fetch-batch-record-count | 2048 | The target number of records for Iceberg reader fetch batch. |
+| table.exec.iceberg.worker-pool-size | ThreadPools.WORKER_THREAD_POOL_SIZE | The size of workers pool used to plan or scan manifests. If the value of Runtime.getRuntime().availableProcessors() is compared to 2, take the larger one. |
+| table.exec.iceberg.use-flip27-source | true | If true, Use the FLIP-27 based Iceberg source implementation. |
Review Comment:
Also this will be deprecated soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org