You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "geserdugarov (via GitHub)" <gi...@apache.org> on 2024/11/02 13:22:19 UTC
[PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
geserdugarov opened a new pull request, #12192:
URL: https://github.com/apache/hudi/pull/12192
### Change Logs
Logic for partition column values parse is removed from `SparkHoodieTableFileIndex`, and localized in `HoodieSparkUtils`.
**Before:**
**After:**
### Impact
No
### Risk level (write none, low medium or high below)
None
### Documentation Update
No need
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453043834
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
}, {
"hash" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481",
"triggerID" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f5a1454ad6a9eda839f99e56c6b5f7186755b7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453010292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbcd92569ad494d16b0da99210683154e313190 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453020069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
}, {
"hash" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481",
"triggerID" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbcd92569ad494d16b0da99210683154e313190 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480)
* 00f5a1454ad6a9eda839f99e56c6b5f7186755b7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12192:
URL: https://github.com/apache/hudi/pull/12192#discussion_r1826910961
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala:
##########
@@ -226,6 +229,31 @@ object HoodieSparkUtils extends SparkAdapterSupport with SparkVersionsSupport wi
sparkAdapter.createSparkRowSerDe(structType)
}
+ def doParsePartitionColumnValues(partitionColumns: Array[String],
Review Comment:
Agree with you, current naming is confusing. Addressed your comment in [7ee1845](https://github.com/apache/hudi/pull/12192/commits/7ee1845ce0c56dbad6216646b565618223ad38a1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453310131
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
}, {
"hash" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481",
"triggerID" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"triggerType" : "PUSH"
}, {
"hash" : "7ee1845ce0c56dbad6216646b565618223ad38a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1483",
"triggerID" : "7ee1845ce0c56dbad6216646b565618223ad38a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f5a1454ad6a9eda839f99e56c6b5f7186755b7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481)
* 7ee1845ce0c56dbad6216646b565618223ad38a1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1483)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453328529
And also it would be great to try to fix this test later:
```
Error: Tests run: 252, Failures: 1, Errors: 0, Skipped: 18, Time elapsed: 1,139.478 s <<< FAILURE! - in JUnit Vintage
Error: [1] tableType=COPY_ON_WRITE(testPartitionStatsWithClustering(HoodieTableType)) Time elapsed: 7.048 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <2> but was: <1>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453309732
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
}, {
"hash" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481",
"triggerID" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"triggerType" : "PUSH"
}, {
"hash" : "7ee1845ce0c56dbad6216646b565618223ad38a1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ee1845ce0c56dbad6216646b565618223ad38a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f5a1454ad6a9eda839f99e56c6b5f7186755b7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481)
* 7ee1845ce0c56dbad6216646b565618223ad38a1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453326824
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
}, {
"hash" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1481",
"triggerID" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"triggerType" : "PUSH"
}, {
"hash" : "7ee1845ce0c56dbad6216646b565618223ad38a1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1483",
"triggerID" : "7ee1845ce0c56dbad6216646b565618223ad38a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ee1845ce0c56dbad6216646b565618223ad38a1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1483)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2452989809
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbcd92569ad494d16b0da99210683154e313190 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12192:
URL: https://github.com/apache/hudi/pull/12192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2452989360
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbcd92569ad494d16b0da99210683154e313190 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453019713
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "afbcd92569ad494d16b0da99210683154e313190",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480",
"triggerID" : "afbcd92569ad494d16b0da99210683154e313190",
"triggerType" : "PUSH"
}, {
"hash" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "00f5a1454ad6a9eda839f99e56c6b5f7186755b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afbcd92569ad494d16b0da99210683154e313190 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1480)
* 00f5a1454ad6a9eda839f99e56c6b5f7186755b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12192:
URL: https://github.com/apache/hudi/pull/12192#discussion_r1826884963
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala:
##########
@@ -226,6 +229,31 @@ object HoodieSparkUtils extends SparkAdapterSupport with SparkVersionsSupport wi
sparkAdapter.createSparkRowSerDe(structType)
}
+ def doParsePartitionColumnValues(partitionColumns: Array[String],
Review Comment:
IMO, we should rename this method to `parsePartitionColumnValues` and `parsePartitionColumnValues` to `doParsePartitionColumnValues`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8455] Logic for partition column values parse is localized in `HoodieSparkUtils` [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12192:
URL: https://github.com/apache/hudi/pull/12192#issuecomment-2453019745
Got:
`TestAsyncCompaction.testConcurrentCompaction:239 compaction plan should not include pending log files ==> expected: <true> but was: <false>`
Possibly, it's a flaky test. Need to check later separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org