You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2023/06/12 03:33:56 UTC
[GitHub] [hudi] SteNicholas opened a new pull request, #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
SteNicholas opened a new pull request, #8933:
URL: https://github.com/apache/hudi/pull/8933
### Change Logs
Spark reads table error when Flink creates table without `precombine.field`. Spark should read table successfully when Flink create table without `precombine.field`.
### Impact
Flink `HoodieCatalog` and `HoodieHiveCatalog` checks `precombine.field` for `createTable`.
### Risk level (write none, low medium or high below)
none.
### Documentation Update
none.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1666396452
Closing it in favor of #9370. That PR handles relaxing precombine and #8107 already hanles for primary key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1594213764
@danny0405, I have addressed above comments. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KenjiFujima commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "KenjiFujima (via GitHub)" <gi...@apache.org>.
KenjiFujima commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1613251280
@danny0405, I have addressed above comments. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8933:
URL: https://github.com/apache/hudi/pull/8933#discussion_r1279045760
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -478,4 +481,36 @@ public static boolean isWriteCommit(HoodieTableType tableType, HoodieInstant ins
public static String getAuxiliaryPath(Configuration conf) {
return conf.getString(FlinkOptions.PATH) + Path.SEPARATOR + AUXILIARYFOLDER_NAME;
}
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param schema The table schema.
+ */
+ public static void checkPreCombineField(Configuration conf, ResolvedSchema schema) {
+ checkPreCombineField(conf, schema.getColumnNames());
+ }
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param columnNames The column names.
+ */
+ public static void checkPreCombineField(Configuration conf, List<String> columnNames) {
+ String preCombineField = conf.get(FlinkOptions.PRECOMBINE_FIELD);
+ if (!columnNames.contains(preCombineField)) {
Review Comment:
@SteNicholas This comment does not look like resolved. Can you please revisit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8933:
URL: https://github.com/apache/hudi/pull/8933#discussion_r1232024093
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieCatalog.java:
##########
@@ -321,21 +318,6 @@ public void createTable(ObjectPath tablePath, CatalogBaseTable catalogTable, boo
options.put(TableOptionProperties.PK_CONSTRAINT_NAME, resolvedSchema.getPrimaryKey().get().getName());
options.put(TableOptionProperties.PK_COLUMNS, pkColumns);
- // check preCombine
- final String preCombineField = conf.getString(FlinkOptions.PRECOMBINE_FIELD);
- if (!resolvedSchema.getColumnNames().contains(preCombineField)) {
- if (OptionsResolver.isDefaultHoodieRecordPayloadClazz(conf)) {
- throw new HoodieValidationException("Option '" + FlinkOptions.PRECOMBINE_FIELD.key()
- + "' is required for payload class: " + DefaultHoodieRecordPayload.class.getName());
Review Comment:
The preCombine field can be omitted only when it is append mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1620169128
@danny0405, could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8933:
URL: https://github.com/apache/hudi/pull/8933#discussion_r1232023559
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java:
##########
@@ -493,10 +493,13 @@ private void initTableIfNotExists(ObjectPath tablePath, CatalogTable catalogTabl
// because the HoodieTableMetaClient is a heavy impl, we try to avoid initializing it
// when calling #getTable.
- if (catalogTable.getUnresolvedSchema().getPrimaryKey().isPresent()
- && !flinkConf.contains(FlinkOptions.RECORD_KEY_FIELD)) {
- final String pkColumns = String.join(",", catalogTable.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
- flinkConf.setString(FlinkOptions.RECORD_KEY_FIELD, pkColumns);
+ if (!flinkConf.contains(FlinkOptions.RECORD_KEY_FIELD)) {
+ if (catalogTable.getUnresolvedSchema().getPrimaryKey().isPresent()) {
+ final String pkColumns = String.join(",", catalogTable.getUnresolvedSchema().getPrimaryKey().get().getColumnNames());
+ flinkConf.setString(FlinkOptions.RECORD_KEY_FIELD, pkColumns);
Review Comment:
Why we must have a primary key definition then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on a diff in pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #8933:
URL: https://github.com/apache/hudi/pull/8933#discussion_r1226086493
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -478,4 +481,36 @@ public static boolean isWriteCommit(HoodieTableType tableType, HoodieInstant ins
public static String getAuxiliaryPath(Configuration conf) {
return conf.getString(FlinkOptions.PATH) + Path.SEPARATOR + AUXILIARYFOLDER_NAME;
}
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param schema The table schema.
+ */
+ public static void checkPreCombineField(Configuration conf, ResolvedSchema schema) {
+ checkPreCombineField(conf, schema.getColumnNames());
+ }
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param columnNames The column names.
+ */
+ public static void checkPreCombineField(Configuration conf, List<String> columnNames) {
+ String preCombineField = conf.get(FlinkOptions.PRECOMBINE_FIELD);
+ if (!columnNames.contains(preCombineField)) {
Review Comment:
@danny0405, it should still throw after spark pkless support #8107, because #8107 is used for auto generation of record keys not precombine field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1586528808
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e05a1ca971e42db6cc274198e0b2702feb0fb6d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1594553086
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
}, {
"hash" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912",
"triggerID" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ab390d9f29c63cdd7a07da37ce1899cb43ce330 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on a diff in pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #8933:
URL: https://github.com/apache/hudi/pull/8933#discussion_r1226086493
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -478,4 +481,36 @@ public static boolean isWriteCommit(HoodieTableType tableType, HoodieInstant ins
public static String getAuxiliaryPath(Configuration conf) {
return conf.getString(FlinkOptions.PATH) + Path.SEPARATOR + AUXILIARYFOLDER_NAME;
}
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param schema The table schema.
+ */
+ public static void checkPreCombineField(Configuration conf, ResolvedSchema schema) {
+ checkPreCombineField(conf, schema.getColumnNames());
+ }
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param columnNames The column names.
+ */
+ public static void checkPreCombineField(Configuration conf, List<String> columnNames) {
+ String preCombineField = conf.get(FlinkOptions.PRECOMBINE_FIELD);
+ if (!columnNames.contains(preCombineField)) {
Review Comment:
@danny0405, it should still throw after spark pkless support #8107, because #8107 is used to auto generation of record keys not precombine field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
URL: https://github.com/apache/hudi/pull/8933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1594271025
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
}, {
"hash" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912",
"triggerID" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e05a1ca971e42db6cc274198e0b2702feb0fb6d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757)
* 9ab390d9f29c63cdd7a07da37ce1899cb43ce330 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1594262058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
}, {
"hash" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e05a1ca971e42db6cc274198e0b2702feb0fb6d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757)
* 9ab390d9f29c63cdd7a07da37ce1899cb43ce330 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1612497373
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
}, {
"hash" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912",
"triggerID" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"triggerType" : "PUSH"
}, {
"hash" : "d1564f421664fd2dee15dfdbdae4dec07baedf92",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18186",
"triggerID" : "d1564f421664fd2dee15dfdbdae4dec07baedf92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ab390d9f29c63cdd7a07da37ce1899cb43ce330 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912)
* d1564f421664fd2dee15dfdbdae4dec07baedf92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18186)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1612491093
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
}, {
"hash" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912",
"triggerID" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"triggerType" : "PUSH"
}, {
"hash" : "d1564f421664fd2dee15dfdbdae4dec07baedf92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d1564f421664fd2dee15dfdbdae4dec07baedf92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ab390d9f29c63cdd7a07da37ce1899cb43ce330 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912)
* d1564f421664fd2dee15dfdbdae4dec07baedf92 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without record key and primary key
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1612806333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
}, {
"hash" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17912",
"triggerID" : "9ab390d9f29c63cdd7a07da37ce1899cb43ce330",
"triggerType" : "PUSH"
}, {
"hash" : "d1564f421664fd2dee15dfdbdae4dec07baedf92",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18186",
"triggerID" : "d1564f421664fd2dee15dfdbdae4dec07baedf92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1564f421664fd2dee15dfdbdae4dec07baedf92 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18186)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1586561271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e05a1ca971e42db6cc274198e0b2702feb0fb6d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8933:
URL: https://github.com/apache/hudi/pull/8933#discussion_r1226058598
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -478,4 +481,36 @@ public static boolean isWriteCommit(HoodieTableType tableType, HoodieInstant ins
public static String getAuxiliaryPath(Configuration conf) {
return conf.getString(FlinkOptions.PATH) + Path.SEPARATOR + AUXILIARYFOLDER_NAME;
}
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param schema The table schema.
+ */
+ public static void checkPreCombineField(Configuration conf, ResolvedSchema schema) {
+ checkPreCombineField(conf, schema.getColumnNames());
+ }
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param columnNames The column names.
+ */
+ public static void checkPreCombineField(Configuration conf, List<String> columnNames) {
+ String preCombineField = conf.get(FlinkOptions.PRECOMBINE_FIELD);
+ if (!columnNames.contains(preCombineField)) {
Review Comment:
Does it still throw after spark pkless support: https://github.com/apache/hudi/pull/8107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8933:
URL: https://github.com/apache/hudi/pull/8933#discussion_r1226289192
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java:
##########
@@ -478,4 +481,36 @@ public static boolean isWriteCommit(HoodieTableType tableType, HoodieInstant ins
public static String getAuxiliaryPath(Configuration conf) {
return conf.getString(FlinkOptions.PATH) + Path.SEPARATOR + AUXILIARYFOLDER_NAME;
}
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param schema The table schema.
+ */
+ public static void checkPreCombineField(Configuration conf, ResolvedSchema schema) {
+ checkPreCombineField(conf, schema.getColumnNames());
+ }
+
+ /**
+ * Check the {@link FlinkOptions#PRECOMBINE_FIELD} of configuration.
+ *
+ * @param conf The flink configuration.
+ * @param columnNames The column names.
+ */
+ public static void checkPreCombineField(Configuration conf, List<String> columnNames) {
+ String preCombineField = conf.get(FlinkOptions.PRECOMBINE_FIELD);
+ if (!columnNames.contains(preCombineField)) {
Review Comment:
After #8107, spark will not throw exception if there is no primary key, maybe we should just fix the primary key set up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8933: [HUDI-5329] Spark reads table error when Flink creates table without precombine.field
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8933:
URL: https://github.com/apache/hudi/pull/8933#issuecomment-1586858990
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757",
"triggerID" : "5e05a1ca971e42db6cc274198e0b2702feb0fb6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e05a1ca971e42db6cc274198e0b2702feb0fb6d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17757)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org