You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Aitozi (via GitHub)" <gi...@apache.org> on 2023/10/10 02:18:19 UTC
[PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Aitozi opened a new pull request, #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104
…ration
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #2037
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1357693778
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
+
+ Map<String, String> optionsFromTableConfig = new HashMap<>();
+
+ ReadableConfig config = context.getConfiguration();
+
+ Map<String, String> conf;
+
+ if (config instanceof Configuration) {
+ conf = ((Configuration) config).toMap();
+ } else if (config instanceof TableConfig) {
+ conf = ((TableConfig) config).getConfiguration().toMap();
+ } else {
+ throw new IllegalArgumentException("Unexpected config: " + config.getClass());
+ }
+
+ String template =
+ String.format(
+ "(%s)\\.(%s|\\*)\\.(%s|\\*)\\.(%s|\\*)\\.(.+)",
+ FlinkConnectorOptions.TABLE_DYNAMIC_OPTION_PREFIX,
+ context.getObjectIdentifier().getCatalogName(),
+ context.getObjectIdentifier().getDatabaseName(),
+ context.getObjectIdentifier().getObjectName());
+ Pattern pattern = Pattern.compile(template);
+
+ conf.keySet()
+ .forEach(
+ (key) -> {
+ Matcher matcher = pattern.matcher(key);
Review Comment:
ok, will do a test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1356119315
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
+
+ Map<String, String> optionsFromTableConfig = new HashMap<>();
+
+ ReadableConfig config = context.getConfiguration();
+
+ Map<String, String> conf;
+
+ if (config instanceof Configuration) {
+ conf = ((Configuration) config).toMap();
+ } else if (config instanceof TableConfig) {
+ conf = ((TableConfig) config).getConfiguration().toMap();
+ } else {
+ throw new IllegalArgumentException("Unexpected config: " + config.getClass());
+ }
+
+ String template =
+ String.format(
+ "(%s)\\.(%s|\\*)\\.(%s|\\*)\\.(%s|\\*)\\.(.+)",
+ FlinkConnectorOptions.TABLE_DYNAMIC_OPTION_PREFIX,
+ context.getObjectIdentifier().getCatalogName(),
+ context.getObjectIdentifier().getDatabaseName(),
+ context.getObjectIdentifier().getObjectName());
+ Pattern pattern = Pattern.compile(template);
+
+ conf.keySet()
+ .forEach(
+ (key) -> {
+ Matcher matcher = pattern.matcher(key);
Review Comment:
It's only happened in client planner phase and the pattern is not complex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1357695553
##########
docs/content/engines/flink.md:
##########
@@ -301,3 +301,19 @@ Users can set memory weight in SQL for Flink Managed Memory, then Flink sink ope
INSERT INTO paimon_table /*+ OPTIONS('sink.use-managed-memory-allocator'='true', 'sink.managed.writer-buffer-memory'='256M') */
SELECT * FROM ....;
```
+## Setting dynamic options
+
+When interacting with the Paimon table, table options can be tuned without changing the options in the catalog. Paimon will extract job-level dynamic options and take effect in the current session.
+The dynamic option's key format is `paimon.${catalogName}.${dbName}.${tableName}.${config_key}`. The catalogName/dbName/tableName can be `*`, which means matching all the specific parts.
+
+For example:
+
+```sql
+-- set scan.timestamp-millis=1697018249000 for the table mycatalog.default.T
+SET 'paimon.mycatalog.default.T.scan.timestamp-millis' = '1697018249000';
+SELECT * FROM T;
+
+-- set scan.timestamp-millis=1697018249000 for the table default.T in any catalog
+SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000';
Review Comment:
> Yes, there is unexpected behavior, but there is also user case to define options for all tables.
if user want to define for all tables they can set `paimon.*.*.*.config_key=value`. This means all tables will take this config_key. if user want to set options under a database, they can set `paimon.*.db.*.config_key=value`. They just have to keep in mind that these three part controls the catalog, database, and table scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1356089979
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
+
+ Map<String, String> optionsFromTableConfig = new HashMap<>();
+
+ ReadableConfig config = context.getConfiguration();
+
+ Map<String, String> conf;
+
+ if (config instanceof Configuration) {
+ conf = ((Configuration) config).toMap();
+ } else if (config instanceof TableConfig) {
+ conf = ((TableConfig) config).getConfiguration().toMap();
+ } else {
+ throw new IllegalArgumentException("Unexpected config: " + config.getClass());
+ }
+
+ String template =
+ String.format(
+ "(%s)\\.(%s|\\*)\\.(%s|\\*)\\.(%s|\\*)\\.(.+)",
+ FlinkConnectorOptions.TABLE_DYNAMIC_OPTION_PREFIX,
+ context.getObjectIdentifier().getCatalogName(),
+ context.getObjectIdentifier().getDatabaseName(),
+ context.getObjectIdentifier().getObjectName());
+ Pattern pattern = Pattern.compile(template);
+
+ conf.keySet()
+ .forEach(
+ (key) -> {
+ Matcher matcher = pattern.matcher(key);
Review Comment:
`pattern.matcher` is very slow, can we just check starts with `paimon.` first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1363726722
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
+
+ Map<String, String> optionsFromTableConfig = new HashMap<>();
+
+ ReadableConfig config = context.getConfiguration();
+
+ Map<String, String> conf;
+
+ if (config instanceof Configuration) {
+ conf = ((Configuration) config).toMap();
+ } else if (config instanceof TableConfig) {
+ conf = ((TableConfig) config).getConfiguration().toMap();
+ } else {
+ throw new IllegalArgumentException("Unexpected config: " + config.getClass());
+ }
+
+ String template =
+ String.format(
+ "(%s)\\.(%s|\\*)\\.(%s|\\*)\\.(%s|\\*)\\.(.+)",
+ FlinkConnectorOptions.TABLE_DYNAMIC_OPTION_PREFIX,
+ context.getObjectIdentifier().getCatalogName(),
+ context.getObjectIdentifier().getDatabaseName(),
+ context.getObjectIdentifier().getObjectName());
+ Pattern pattern = Pattern.compile(template);
+
+ conf.keySet()
+ .forEach(
+ (key) -> {
+ Matcher matcher = pattern.matcher(key);
Review Comment:
OpenJDK 64-Bit Server VM 1.8.0_292-b10 on Mac OS X 10.16
Apple M1 Pro
test-pattern: Best/Avg Time(ms) Row Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
OPERATORTEST_test-pattern_pattern 12530 / 12779 0.0 41767.0 1.0X
OPERATORTEST_test-pattern_startswith 500 / 538 0.6 1666.2 25.1X
Per row will run for 100 key. so the max cost for 100 key match is 41767 ns. The startsWith is much faster, but I think both of the cost is acceptable, and the startWith check is not flexible when we only want to set for the specific table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1352580842
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
Review Comment:
Can you also document this in flink page?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
Review Comment:
Can you also document this in engine/flink page?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1357665247
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
+
+ Map<String, String> optionsFromTableConfig = new HashMap<>();
+
+ ReadableConfig config = context.getConfiguration();
+
+ Map<String, String> conf;
+
+ if (config instanceof Configuration) {
+ conf = ((Configuration) config).toMap();
+ } else if (config instanceof TableConfig) {
+ conf = ((TableConfig) config).getConfiguration().toMap();
+ } else {
+ throw new IllegalArgumentException("Unexpected config: " + config.getClass());
+ }
+
+ String template =
+ String.format(
+ "(%s)\\.(%s|\\*)\\.(%s|\\*)\\.(%s|\\*)\\.(.+)",
+ FlinkConnectorOptions.TABLE_DYNAMIC_OPTION_PREFIX,
+ context.getObjectIdentifier().getCatalogName(),
+ context.getObjectIdentifier().getDatabaseName(),
+ context.getObjectIdentifier().getObjectName());
+ Pattern pattern = Pattern.compile(template);
+
+ conf.keySet()
+ .forEach(
+ (key) -> {
+ Matcher matcher = pattern.matcher(key);
Review Comment:
You can do a little benchmark for these two methods, and I think there may be hundreds options need to be checked.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
+
+ Map<String, String> optionsFromTableConfig = new HashMap<>();
+
+ ReadableConfig config = context.getConfiguration();
+
+ Map<String, String> conf;
+
+ if (config instanceof Configuration) {
+ conf = ((Configuration) config).toMap();
+ } else if (config instanceof TableConfig) {
+ conf = ((TableConfig) config).getConfiguration().toMap();
+ } else {
+ throw new IllegalArgumentException("Unexpected config: " + config.getClass());
+ }
+
+ String template =
+ String.format(
+ "(%s)\\.(%s|\\*)\\.(%s|\\*)\\.(%s|\\*)\\.(.+)",
+ FlinkConnectorOptions.TABLE_DYNAMIC_OPTION_PREFIX,
+ context.getObjectIdentifier().getCatalogName(),
+ context.getObjectIdentifier().getDatabaseName(),
+ context.getObjectIdentifier().getObjectName());
+ Pattern pattern = Pattern.compile(template);
+
+ conf.keySet()
+ .forEach(
+ (key) -> {
+ Matcher matcher = pattern.matcher(key);
Review Comment:
You can do a little benchmark for these two methods, and I think there may be hundreds options need to be check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1357665828
##########
docs/content/engines/flink.md:
##########
@@ -301,3 +301,19 @@ Users can set memory weight in SQL for Flink Managed Memory, then Flink sink ope
INSERT INTO paimon_table /*+ OPTIONS('sink.use-managed-memory-allocator'='true', 'sink.managed.writer-buffer-memory'='256M') */
SELECT * FROM ....;
```
+## Setting dynamic options
+
+When interacting with the Paimon table, table options can be tuned without changing the options in the catalog. Paimon will extract job-level dynamic options and take effect in the current session.
+The dynamic option's key format is `paimon.${catalogName}.${dbName}.${tableName}.${config_key}`. The catalogName/dbName/tableName can be `*`, which means matching all the specific parts.
+
+For example:
+
+```sql
+-- set scan.timestamp-millis=1697018249000 for the table mycatalog.default.T
+SET 'paimon.mycatalog.default.T.scan.timestamp-millis' = '1697018249000';
+SELECT * FROM T;
+
+-- set scan.timestamp-millis=1697018249000 for the table default.T in any catalog
+SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000';
Review Comment:
Maybe not `scan.timestamp-millis`, but some options about execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1357695553
##########
docs/content/engines/flink.md:
##########
@@ -301,3 +301,19 @@ Users can set memory weight in SQL for Flink Managed Memory, then Flink sink ope
INSERT INTO paimon_table /*+ OPTIONS('sink.use-managed-memory-allocator'='true', 'sink.managed.writer-buffer-memory'='256M') */
SELECT * FROM ....;
```
+## Setting dynamic options
+
+When interacting with the Paimon table, table options can be tuned without changing the options in the catalog. Paimon will extract job-level dynamic options and take effect in the current session.
+The dynamic option's key format is `paimon.${catalogName}.${dbName}.${tableName}.${config_key}`. The catalogName/dbName/tableName can be `*`, which means matching all the specific parts.
+
+For example:
+
+```sql
+-- set scan.timestamp-millis=1697018249000 for the table mycatalog.default.T
+SET 'paimon.mycatalog.default.T.scan.timestamp-millis' = '1697018249000';
+SELECT * FROM T;
+
+-- set scan.timestamp-millis=1697018249000 for the table default.T in any catalog
+SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000';
Review Comment:
> Yes, there is unexpected behavior, but there is also user case to define options for all tables.
if user want to define for all tables they can set `paimon.*.*.*.config_key=value`. This means all tables will take this config_key. if user want to set options for table in a database, they can set `paimon.*.db.*.config_key=value`. They just have to keep in mind that these three part controls the catalog, database, and table scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1357665599
##########
docs/content/engines/flink.md:
##########
@@ -301,3 +301,19 @@ Users can set memory weight in SQL for Flink Managed Memory, then Flink sink ope
INSERT INTO paimon_table /*+ OPTIONS('sink.use-managed-memory-allocator'='true', 'sink.managed.writer-buffer-memory'='256M') */
SELECT * FROM ....;
```
+## Setting dynamic options
+
+When interacting with the Paimon table, table options can be tuned without changing the options in the catalog. Paimon will extract job-level dynamic options and take effect in the current session.
+The dynamic option's key format is `paimon.${catalogName}.${dbName}.${tableName}.${config_key}`. The catalogName/dbName/tableName can be `*`, which means matching all the specific parts.
+
+For example:
+
+```sql
+-- set scan.timestamp-millis=1697018249000 for the table mycatalog.default.T
+SET 'paimon.mycatalog.default.T.scan.timestamp-millis' = '1697018249000';
+SELECT * FROM T;
+
+-- set scan.timestamp-millis=1697018249000 for the table default.T in any catalog
+SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000';
Review Comment:
Yes, there is unexpected behavior, but there is also user case to define options for all tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1356109349
##########
docs/content/engines/flink.md:
##########
@@ -301,3 +301,19 @@ Users can set memory weight in SQL for Flink Managed Memory, then Flink sink ope
INSERT INTO paimon_table /*+ OPTIONS('sink.use-managed-memory-allocator'='true', 'sink.managed.writer-buffer-memory'='256M') */
SELECT * FROM ....;
```
+## Setting dynamic options
+
+When interacting with the Paimon table, table options can be tuned without changing the options in the catalog. Paimon will extract job-level dynamic options and take effect in the current session.
+The dynamic option's key format is `paimon.${catalogName}.${dbName}.${tableName}.${config_key}`. The catalogName/dbName/tableName can be `*`, which means matching all the specific parts.
+
+For example:
+
+```sql
+-- set scan.timestamp-millis=1697018249000 for the table mycatalog.default.T
+SET 'paimon.mycatalog.default.T.scan.timestamp-millis' = '1697018249000';
+SELECT * FROM T;
+
+-- set scan.timestamp-millis=1697018249000 for the table default.T in any catalog
+SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000';
Review Comment:
for all table may cause unexpected behavior, if we use two table but only want to set option to one of them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1354765646
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/AbstractFlinkTableFactory.java:
##########
@@ -263,4 +286,57 @@ static boolean schemaEquals(RowType rowType1, RowType rowType2) {
}
return true;
}
+
+ /**
+ * The dynamic option's format is:
+ *
+ * <p>{@link
+ * FlinkConnectorOptions#TABLE_DYNAMIC_OPTION_PREFIX}.${catalog}.${database}.${tableName}.key =
+ * value. These job level configs will be extracted and injected into the target table option.
+ *
+ * @param context The table factory context.
+ * @return The dynamic options of this target table.
+ */
+ static Map<String, String> getDynamicTableConfigOptions(DynamicTableFactory.Context context) {
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#discussion_r1356090535
##########
docs/content/engines/flink.md:
##########
@@ -301,3 +301,19 @@ Users can set memory weight in SQL for Flink Managed Memory, then Flink sink ope
INSERT INTO paimon_table /*+ OPTIONS('sink.use-managed-memory-allocator'='true', 'sink.managed.writer-buffer-memory'='256M') */
SELECT * FROM ....;
```
+## Setting dynamic options
+
+When interacting with the Paimon table, table options can be tuned without changing the options in the catalog. Paimon will extract job-level dynamic options and take effect in the current session.
+The dynamic option's key format is `paimon.${catalogName}.${dbName}.${tableName}.${config_key}`. The catalogName/dbName/tableName can be `*`, which means matching all the specific parts.
+
+For example:
+
+```sql
+-- set scan.timestamp-millis=1697018249000 for the table mycatalog.default.T
+SET 'paimon.mycatalog.default.T.scan.timestamp-millis' = '1697018249000';
+SELECT * FROM T;
+
+-- set scan.timestamp-millis=1697018249000 for the table default.T in any catalog
+SET 'paimon.*.default.T.scan.timestamp-millis' = '1697018249000';
Review Comment:
Can we just support paimon.{key} for all tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [flink] Support to set the table option by dynamic global job configu… [incubator-paimon]
Posted by "Aitozi (via GitHub)" <gi...@apache.org>.
Aitozi commented on PR #2104:
URL: https://github.com/apache/incubator-paimon/pull/2104#issuecomment-1769795252
Updated, Please take a look again @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org