You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/29 21:39:21 UTC
[GitHub] [hudi] yihua opened a new pull request, #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
yihua opened a new pull request, #6250:
URL: https://github.com/apache/hudi/pull/6250
## What is the purpose of the pull request
This PR improves file name extraction logic in metadata utils by adding a new util method.
## Brief change log
- Adds a new util method `FSUtils.getFileName`
- Refactors the logic of extracting file names in `HoodieTableMetadataUtil`
- Adds a unit test for the new util method
## Verify this pull request
This change adds a new test as mentioned above.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6250:
URL: https://github.com/apache/hudi/pull/6250#discussion_r933634375
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1162,13 +1154,8 @@ private static Stream<HoodieRecord> getColumnStatsRecords(String partitionPath,
HoodieTableMetaClient datasetMetaClient,
List<String> columnsToIndex,
boolean isDeleted) {
- String partitionName = getPartitionIdentifier(partitionPath);
- // NOTE: We have to chop leading "/" to make sure Hadoop does not treat it like
- // absolute path
String filePartitionPath = filePath.startsWith("/") ? filePath.substring(1) : filePath;
- String fileName = partitionName.equals(NON_PARTITIONED_NAME)
- ? filePartitionPath
- : filePartitionPath.substring(partitionName.length() + 1);
+ String fileName = FSUtils.getFileName(filePath, partitionPath);
Review Comment:
The same here, using `partitionPath` directly instead of the partition identified.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -325,16 +325,13 @@ public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCo
return map;
}
- int offset = partition.equals(NON_PARTITIONED_NAME)
- ? (pathWithPartition.startsWith("/") ? 1 : 0)
- : partition.length() + 1;
- String filename = pathWithPartition.substring(offset);
+ String fileName = FSUtils.getFileName(pathWithPartition, partitionStatName);
Review Comment:
Before the change, the `partition` identifier is used, instead of `partitionStatName`. For a partitioned table, there is no difference; for a non-partitioned table, the `partition` identifier is `.` while `partitionStatName` could be empty or `/`. The new logic depends on `partitionStatName` instead of `partition` identified, and the file name extracted is not affected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6250:
URL: https://github.com/apache/hudi/pull/6250#discussion_r933696374
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -325,16 +325,13 @@ public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCo
return map;
}
- int offset = partition.equals(NON_PARTITIONED_NAME)
- ? (pathWithPartition.startsWith("/") ? 1 : 0)
- : partition.length() + 1;
- String filename = pathWithPartition.substring(offset);
+ String fileName = FSUtils.getFileName(pathWithPartition, partitionStatName);
Review Comment:
the new util getFileName() uses `partition`, right? why not pass `partition` to check? as how previously done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6250:
URL: https://github.com/apache/hudi/pull/6250#issuecomment-1199968512
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464",
"triggerID" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 129007ab3840f01ccafaf5ef73275301fcd6799f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6250:
URL: https://github.com/apache/hudi/pull/6250#issuecomment-1200016266
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464",
"triggerID" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 129007ab3840f01ccafaf5ef73275301fcd6799f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6250:
URL: https://github.com/apache/hudi/pull/6250#discussion_r933697770
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -325,16 +325,13 @@ public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCo
return map;
}
- int offset = partition.equals(NON_PARTITIONED_NAME)
- ? (pathWithPartition.startsWith("/") ? 1 : 0)
- : partition.length() + 1;
- String filename = pathWithPartition.substring(offset);
+ String fileName = FSUtils.getFileName(pathWithPartition, partitionStatName);
Review Comment:
ok i see. i was confused by the var name `partition`, which should actually be called `partitionIdentifier`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6250:
URL: https://github.com/apache/hudi/pull/6250#issuecomment-1200066863
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464",
"triggerID" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 129007ab3840f01ccafaf5ef73275301fcd6799f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6250:
URL: https://github.com/apache/hudi/pull/6250#issuecomment-1200057185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464",
"triggerID" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 129007ab3840f01ccafaf5ef73275301fcd6799f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10464)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6250:
URL: https://github.com/apache/hudi/pull/6250#issuecomment-1200056238
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 129007ab3840f01ccafaf5ef73275301fcd6799f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #6250:
URL: https://github.com/apache/hudi/pull/6250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6250:
URL: https://github.com/apache/hudi/pull/6250#issuecomment-1199966586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "129007ab3840f01ccafaf5ef73275301fcd6799f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 129007ab3840f01ccafaf5ef73275301fcd6799f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6250: [HUDI-4507] Improve file name extraction logic in metadata utils
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6250:
URL: https://github.com/apache/hudi/pull/6250#discussion_r933697289
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -325,16 +325,13 @@ public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCo
return map;
}
- int offset = partition.equals(NON_PARTITIONED_NAME)
- ? (pathWithPartition.startsWith("/") ? 1 : 0)
- : partition.length() + 1;
- String filename = pathWithPartition.substring(offset);
+ String fileName = FSUtils.getFileName(pathWithPartition, partitionStatName);
Review Comment:
We cannot use `partition` here which is generated by `getPartitionIdentifier(partitionStatName)`, changing the empty relative partition path to `.` partition identifier. `getFileName()` expects plain relative partition path, instead of the partition identifier used in the metadata table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org