You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/28 08:09:12 UTC
[GitHub] [hudi] onlywangyh opened a new pull request, #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
onlywangyh opened a new pull request, #5434:
URL: https://github.com/apache/hudi/pull/5434
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
fix https://github.com/apache/hudi/issues/5394
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113000880
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d16bea801086257ae640e5b8cef7a669dc51ad9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379)
* d1c01449a9458110609fe93b06fa8915acac8124 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1109464654
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext this PARTITION_PATH_FIELD assign to hive sync partition fields .I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values of the two will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is same as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. We can't get the datetime value from this hive table, and the table partition is also broken. This datetime value is conflicting
When we set PARTITION PATH_FIELD value is different with HIVE_SYNC PARTITION FIELDS value.
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=inc_day
`
We can get this table like ：
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`inc_day` string COMMENT '')...
```
In this time we can normal get the _datetime_ value, and the _inc_day_ as a partition field is also work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r860472525
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/Pipelines.java:
##########
@@ -120,7 +120,7 @@ public static DataStreamSink<Object> bulkInsert(Configuration conf, RowType rowT
.name("dummy");
}
- final String[] partitionFields = FilePathUtils.extractPartitionKeys(conf);
+ final String[] partitionFields = FilePathUtils.extractHivePartitionFields(conf);
Review Comment:
You should not change this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111780522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2897aacab86dcd0ebe26cda0f755d86e7d414da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352)
* fdb36e22aab32707f40ac0f846e05073723b5f9d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112903836
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3d16bea801086257ae640e5b8cef7a669dc51ad9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112315351
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111881858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18c7ef7778de10a9156908953f0146cc9b0467cc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112812588
@hudi-bot run azur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1109604884
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113268570
This compile has passed in local. But the Azure build [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385) .
> I get the build error.
>
> ```
> [ERROR] Errors:
> [ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
> [ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
> [ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
> [ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
> [ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
> [ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
> [ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
> [ERROR] TestKafkaAvroSchemaDeserializer>UtilitiesTestBase.setup:156 » Spark Only one S...
> ```
>
> This `UtilitiesTestBase.setup` is not change what's problem here?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext this PARTITION_PATH_FIELD assign to hive sync partition fields .I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values of the two will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is same as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. The problems is we can't get the datetime from hive And the partition is also broken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111891647
> * @hudi-bot run azure
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111964306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18c7ef7778de10a9156908953f0146cc9b0467cc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358)
* 736bb17c069c67bf26e4313d5331fcd21c559c6e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111746640
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2897aacab86dcd0ebe26cda0f755d86e7d414da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112174222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 736bb17c069c67bf26e4313d5331fcd21c559c6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112181434
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 736bb17c069c67bf26e4313d5331fcd21c559c6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367)
* a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112177898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 736bb17c069c67bf26e4313d5331fcd21c559c6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367)
* a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112319253
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368)
* f4084a5734d579888aa29a72df25de3e9521b67e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext this PARTITION_PATH_FIELD assign to hive sync partition fields .I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is same as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. We can't get the datetime value from this hive table, and the table partition is also broken. This datetime value is conflicting
When we set PARTITION PATH_FIELD value is different with HIVE_SYNC PARTITION FIELDS value like this.
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=inc_day
`
We can get this table like ：
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`inc_day` string COMMENT '')...
```
In this time we can normal get the _datetime_ value, and the _inc_day_ as a partition field is also work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111878476
@onlywangyh You may need to fix the build failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111892672
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values of the two will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is smae as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. The problems is we can't get the datetime from hive And the partition is also broken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext this PARTITION_PATH_FIELD assign to hive sync partition fields .I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values of the two will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is same as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. You can see we can't get the datetime value from hive And the partition is also broken. This datetime value is conflicting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext this PARTITION_PATH_FIELD assign to hive sync partition fields .I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values of the two will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is same as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. We can't get the datetime value from this hive table, and the table partition is also broken. This datetime value is conflicting
When we set PARTITION PATH_FIELD value is different with HIVE_SYNC PARTITION FIELDS value like this.
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=inc_day
`
We can get this table like ：
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`inc_day` string COMMENT '')...
```
In this time we can normal get the _datetime_ value, and the _inc_day_ as a partition field is also work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111875203
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdb36e22aab32707f40ac0f846e05073723b5f9d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356)
* 18c7ef7778de10a9156908953f0146cc9b0467cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh closed pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh closed pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
URL: https://github.com/apache/hudi/pull/5434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
yihua commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113601309
> This compile has passed in local. But the Azure build [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385) .
> >
> > This `UtilitiesTestBase.setup` is not change what's problem here?.
This is due to CI failure on master. I've fixed the issue. I rebased this PR and pushed the updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111872185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdb36e22aab32707f40ac0f846e05073723b5f9d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356)
* 18c7ef7778de10a9156908953f0146cc9b0467cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113882082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
}, {
"hash" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394",
"triggerID" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"triggerType" : "PUSH"
}, {
"hash" : "8f9e73dc3f4a2d0900438e0a10bf079e837934c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8f9e73dc3f4a2d0900438e0a10bf079e837934c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e2055530bfe4c1cdace661801b3fbc45a22caf8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394)
* 8f9e73dc3f4a2d0900438e0a10bf079e837934c4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r860472711
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/HiveSyncContextTest.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.hudi.sink.utils;
+
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import org.apache.flink.configuration.Configuration;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Test cases for {@link HiveSyncContext}.
+ */
+public class HiveSyncContextTest {
+ /**
Review Comment:
nit: class name -> TestHiveSyncContext
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111877973
> LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r859325849
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
Got you, then in `HiveSyncContext`, you can take the option `HIVE_SYNC_PARTITION_FIELDS` instead, and write a new tool method(you should not modify the existing method directly), if user does not configure the `HIVE_SYNC_PARTITION_FIELDS`, fallback to hudi table's partition path (the current code does so).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112996802
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 3d16bea801086257ae640e5b8cef7a669dc51ad9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112325378
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368)
* f4084a5734d579888aa29a72df25de3e9521b67e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111691407
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321)
* d2897aacab86dcd0ebe26cda0f755d86e7d414da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111689652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321)
* d2897aacab86dcd0ebe26cda0f755d86e7d414da UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113882859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
}, {
"hash" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394",
"triggerID" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"triggerType" : "PUSH"
}, {
"hash" : "8f9e73dc3f4a2d0900438e0a10bf079e837934c4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8398",
"triggerID" : "8f9e73dc3f4a2d0900438e0a10bf079e837934c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e2055530bfe4c1cdace661801b3fbc45a22caf8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394)
* 8f9e73dc3f4a2d0900438e0a10bf079e837934c4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1109467962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112958422
@hudi-bot run azur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112805876
Please fix the build failure, you can rebase the latest master and force push using command:
```git
git fetch upstream master
git rebase upstream/master
git push origin {branch_name} --force
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r860567134
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestHiveSyncContext.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.hudi.sink.utils;
+
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import org.apache.flink.configuration.Configuration;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Test cases for {@link HiveSyncContext}.
+ */
+public class TestHiveSyncContext {
+ /**
+ * Test that the file ids generated by the task can finally shuffled to itself.
+ */
+ @Test
+ void testBuildSyncConfig() throws ClassNotFoundException, NoSuchMethodException, InvocationTargetException, IllegalAccessException {
+ Configuration configuration1 = new Configuration();
+ Configuration configuration2 = new Configuration();
+ String hiveSyncPartitionField = "hiveSyncPartitionField";
+ String partitionPathField = "partitionPathField";
+
+ configuration1.setString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS, hiveSyncPartitionField);
+ configuration1.setString(FlinkOptions.PARTITION_PATH_FIELD, partitionPathField);
+
+ configuration2.setString(FlinkOptions.PARTITION_PATH_FIELD, partitionPathField);
+
+ Class<?> threadClazz = Class.forName("org.apache.hudi.sink.utils.HiveSyncContext");
+ Method buildSyncConfigMethod = threadClazz.getDeclaredMethod("buildSyncConfig", Configuration.class);
Review Comment:
Usually we do not use the reflection to make method visible, you can just make the method public and tag it with annotation
```java
@VisibleForTesting
```
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/utils/TestHiveSyncContext.java:
##########
@@ -0,0 +1,47 @@
+package org.apache.hudi.sink.utils;
+
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import org.apache.flink.configuration.Configuration;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Test cases for {@link HiveSyncContext}.
+ */
+public class TestHiveSyncContext {
+ /**
+ * Test that the file ids generated by the task can finally shuffled to itself.
+ */
+ @Test
+ void testBuildSyncConfig() throws ClassNotFoundException, NoSuchMethodException, InvocationTargetException, IllegalAccessException {
+ Configuration configuration1 = new Configuration();
Review Comment:
`throws Exception` is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111782622
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2897aacab86dcd0ebe26cda0f755d86e7d414da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352)
* fdb36e22aab32707f40ac0f846e05073723b5f9d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111836142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fdb36e22aab32707f40ac0f846e05073723b5f9d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113634246
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
}, {
"hash" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385)
* 9e2055530bfe4c1cdace661801b3fbc45a22caf8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext this PARTITION_PATH_FIELD assign to hive sync partition fields .I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values of the two will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is smae as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. The problems is we can't get the datetime from hive And the partition is also broken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113085438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1c01449a9458110609fe93b06fa8915acac8124 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383)
* 04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111998524
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 18c7ef7778de10a9156908953f0146cc9b0467cc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358)
* 736bb17c069c67bf26e4313d5331fcd21c559c6e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112163057
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112446915
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f4084a5734d579888aa29a72df25de3e9521b67e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r860472455
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/OptionsResolver.java:
##########
@@ -99,7 +99,7 @@ public static boolean isDeltaTimeCompaction(Configuration conf) {
* Returns whether the table is partitioned.
*/
public static boolean isPartitionedTable(Configuration conf) {
- return FilePathUtils.extractPartitionKeys(conf).length > 0;
+ return FilePathUtils.extractHivePartitionFields(conf).length > 0;
}
Review Comment:
You should not change this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
yihua commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113792789
@onlywangyh There are new failures:
```
[ERROR] Failures:
[ERROR] org.apache.hudi.table.action.compact.TestInlineCompaction.testCompactionRetryOnFailureBasedOnTime
[ERROR] Run 1: TestInlineCompaction.testCompactionRetryOnFailureBasedOnTime:234 expected: <4> but was: <5>
[ERROR] Run 2: TestInlineCompaction.testCompactionRetryOnFailureBasedOnTime:234 expected: <4> but was: <5>
[ERROR] Run 3: TestInlineCompaction.testCompactionRetryOnFailureBasedOnTime:234 expected: <4> but was: <5>
[ERROR] Run 4: TestInlineCompaction.testCompactionRetryOnFailureBasedOnTime:234 expected: <4> but was: <5>
[INFO]
[WARNING] Flakes:
[WARNING] org.apache.hudi.table.action.commit.TestCopyOnWriteActionExecutor.testPartitionMetafileFormat(boolean)[2]
[ERROR] Run 1: TestCopyOnWriteActionExecutor.testPartitionMetafileFormat:536 expected: <true> but was: <false>
[INFO] Run 2: PASS
[INFO]
[INFO]
[ERROR] Tests run: 372, Failures: 1, Errors: 0, Skipped: 0, Flakes: 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113900357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
}, {
"hash" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394",
"triggerID" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"triggerType" : "PUSH"
}, {
"hash" : "8f9e73dc3f4a2d0900438e0a10bf079e837934c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8398",
"triggerID" : "8f9e73dc3f4a2d0900438e0a10bf079e837934c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8f9e73dc3f4a2d0900438e0a10bf079e837934c4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8398)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113961647
Thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r859326135
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
You also need to write a test case i guess. A unit test for HiveSyncContext is okey i think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1111935606
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 18c7ef7778de10a9156908953f0146cc9b0467cc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112071774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 736bb17c069c67bf26e4313d5331fcd21c559c6e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112308752
I get the build error.
```
[ERROR] Errors:
[ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
[ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
[ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
[ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
[ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
[ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
[ERROR] TestSchemaPostProcessor>UtilitiesTestBase.setup:156 » Spark Only one SparkCont...
[ERROR] TestKafkaAvroSchemaDeserializer>UtilitiesTestBase.setup:156 » Spark Only one S...
```
This `UtilitiesTestBase.setup` is not change what's problem here?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112813546
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 3d16bea801086257ae640e5b8cef7a669dc51ad9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112814736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 3d16bea801086257ae640e5b8cef7a669dc51ad9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1112914520
There are still build error, can you fix that :) you can build if first locally to make sure the compile pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113004532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d16bea801086257ae640e5b8cef7a669dc51ad9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379)
* d1c01449a9458110609fe93b06fa8915acac8124 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113081976
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3d16bea801086257ae640e5b8cef7a669dc51ad9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379)
* d1c01449a9458110609fe93b06fa8915acac8124 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383)
* 04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113176257
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858431018
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext, we already assign the partition fields to to hive sync partition fields, see:
https://github.com/apache/hudi/blob/762623a15cfeba6f3fe936c238d660685ae62b50/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/HiveSyncContext.java#L86
so what's the problem here ? BTW, this is a common util which you should not modify directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] onlywangyh commented on a diff in pull request #5434: fix hudi use partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
onlywangyh commented on code in PR #5434:
URL: https://github.com/apache/hudi/pull/5434#discussion_r858555712
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FilePathUtils.java:
##########
@@ -420,9 +420,9 @@ public static org.apache.flink.core.fs.Path toFlinkPath(Path path) {
* @return array of the partition fields
*/
public static String[] extractPartitionKeys(org.apache.flink.configuration.Configuration conf) {
- if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.PARTITION_PATH_FIELD)) {
+ if (FlinkOptions.isDefaultValueDefined(conf, FlinkOptions.HIVE_SYNC_PARTITION_FIELDS)) {
return new String[0];
}
- return conf.getString(FlinkOptions.PARTITION_PATH_FIELD).split(",");
+ return conf.getString(FlinkOptions.HIVE_SYNC_PARTITION_FIELDS).split(",");
}
Review Comment:
In HiveSyncContext this PARTITION_PATH_FIELD assign to hive sync partition fields .I think these two params `PARTITION_PATH_FIELD` and `HIVE_SYNC_PARTITION_FIELDS`
have different meanings in hudi.
`PARTITION_PATH_FIELD` is for hudi KeyGenerator to get a partitionPath
`HIVE_SYNC_PARTITION_FIELDS` is use for hive to set a partition field.
This function _extractPartitionKeys_ should get the hive partition fields key rather than a hudi partition path field. Sometimes confuse the values of the two will cause some errors
In this case we use TimestampBasedAvroKeyGenerator and set hudi partition path field is same as hive partition fields . There will be some promblems, see:
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=datetime
`
**In hudi:** we will get the _1596074902000L_ value and converted to a string hudi partition path like _2020-07-30_.
**In hive:** We will get the table like :
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`datetime` string COMMENT '')...
```
This partition value _datetime=2020-07-30_ also will be add to hive. You can see we can't get the datetime value from hive And the partition is also broken. This datetime value is conflicting
When we set PARTITION PATH_FIELD value is different with HIVE_SYNC PARTITION FIELDS value.
`
PARTITION_PATH_FIELD=datetime
HIVE_SYNC_PARTITION_FIELDS=inc_day
`
We can get this table like ：
```
CREATE EXTERNAL TABLE `testTable`(
`_hoodie_commit_time` string COMMENT '',
`_hoodie_commit_seqno` string COMMENT '',
`_hoodie_record_key` string COMMENT '',
`_hoodie_partition_path` string COMMENT '',
`_hoodie_file_name` string COMMENT '',
`id` int COMMENT '',
`datetime` bigint COMMENT ''
)
PARTITIONED BY (`inc_day` string COMMENT '')...
```
In this time we can normal get the _datetime_ value, and the _inc_day_ as a partition field is also work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113637119
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
}, {
"hash" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394",
"triggerID" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385)
* 9e2055530bfe4c1cdace661801b3fbc45a22caf8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5434:
URL: https://github.com/apache/hudi/pull/5434#issuecomment-1113730291
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8321",
"triggerID" : "a0d19c3725f5c5dc950a55fac3fb26ab62dc7d24",
"triggerType" : "PUSH"
}, {
"hash" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8352",
"triggerID" : "d2897aacab86dcd0ebe26cda0f755d86e7d414da",
"triggerType" : "PUSH"
}, {
"hash" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8356",
"triggerID" : "fdb36e22aab32707f40ac0f846e05073723b5f9d",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8357",
"triggerID" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"triggerType" : "PUSH"
}, {
"hash" : "18c7ef7778de10a9156908953f0146cc9b0467cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8358",
"triggerID" : "1111892672",
"triggerType" : "MANUAL"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8359",
"triggerID" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"triggerType" : "PUSH"
}, {
"hash" : "736bb17c069c67bf26e4313d5331fcd21c559c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8367",
"triggerID" : "1112163057",
"triggerType" : "MANUAL"
}, {
"hash" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8368",
"triggerID" : "a5c34f02df7d69ea17d9b7eb18c4f1264ce3fe8a",
"triggerType" : "PUSH"
}, {
"hash" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8370",
"triggerID" : "f4084a5734d579888aa29a72df25de3e9521b67e",
"triggerType" : "PUSH"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1112812588",
"triggerType" : "MANUAL"
}, {
"hash" : "3d16bea801086257ae640e5b8cef7a669dc51ad9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8379",
"triggerID" : "1112958422",
"triggerType" : "MANUAL"
}, {
"hash" : "d1c01449a9458110609fe93b06fa8915acac8124",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8383",
"triggerID" : "d1c01449a9458110609fe93b06fa8915acac8124",
"triggerType" : "PUSH"
}, {
"hash" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8385",
"triggerID" : "04b172fb4efb9ba0ae7f3f5bac0d34cc0811d571",
"triggerType" : "PUSH"
}, {
"hash" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394",
"triggerID" : "9e2055530bfe4c1cdace661801b3fbc45a22caf8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e2055530bfe4c1cdace661801b3fbc45a22caf8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8394)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #5434: [HUDI-3978] Fix use of partition path field as hive partition field in flink
Posted by GitBox <gi...@apache.org>.
yihua merged PR #5434:
URL: https://github.com/apache/hudi/pull/5434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org