You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/01 10:11:23 UTC
[GitHub] [hudi] TJX2014 opened a new pull request, #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
TJX2014 opened a new pull request, #7105:
URL: https://github.com/apache/hudi/pull/7105
### Change Logs
Make getFileSystem way unify to org.apache.hudi.common.fs.FSUtils#getFs(org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration)
### Impact
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TJX2014 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
TJX2014 commented on code in PR #7105:
URL: https://github.com/apache/hudi/pull/7105#discussion_r1011105643
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java:
##########
@@ -79,7 +79,7 @@ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFi
result.addAll(context.flatMap(subDirectories, directory -> {
PathFilter pathFilter = getFilePathFilter(baseFileExtension);
Path path = new Path(directory);
- FileSystem fileSystem = path.getFileSystem(new Configuration());
+ FileSystem fileSystem = FSUtils.getFs(path, fs.getConf());
RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true);
Review Comment:
For context.flatMap is not thread safe, three is many thread in `context.flatMap` may operate same `fs` if not create new one in `directory -> {} function`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-2335088153
@danny0405 is this still a bug or we can close the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TJX2014 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
TJX2014 commented on code in PR #7105:
URL: https://github.com/apache/hudi/pull/7105#discussion_r1012468649
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java:
##########
@@ -79,7 +79,7 @@ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFi
result.addAll(context.flatMap(subDirectories, directory -> {
PathFilter pathFilter = getFilePathFilter(baseFileExtension);
Path path = new Path(directory);
- FileSystem fileSystem = path.getFileSystem(new Configuration());
+ FileSystem fileSystem = FSUtils.getFs(path, fs.getConf());
RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true);
Review Comment:
So we just remove `path.getFileSystem(new Configuration());`, and rename `fs` ouside to `filesystem` in lambda，right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315029694
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
}, {
"hash" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022",
"triggerID" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728)
* 4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022)
* ace488c3ed6ec5b91594a544932d5b4fb8ceddd5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7105:
URL: https://github.com/apache/hudi/pull/7105#discussion_r1012471254
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java:
##########
@@ -79,7 +79,7 @@ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFi
result.addAll(context.flatMap(subDirectories, directory -> {
PathFilter pathFilter = getFilePathFilter(baseFileExtension);
Path path = new Path(directory);
- FileSystem fileSystem = path.getFileSystem(new Configuration());
+ FileSystem fileSystem = FSUtils.getFs(path, fs.getConf());
RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true);
Review Comment:
I guess it though ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1298532248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84265e2e3f1c60d130fdd50be33e852f76386a0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7105:
URL: https://github.com/apache/hudi/pull/7105#discussion_r1012447758
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java:
##########
@@ -79,7 +79,7 @@ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFi
result.addAll(context.flatMap(subDirectories, directory -> {
PathFilter pathFilter = getFilePathFilter(baseFileExtension);
Path path = new Path(directory);
- FileSystem fileSystem = path.getFileSystem(new Configuration());
+ FileSystem fileSystem = FSUtils.getFs(path, fs.getConf());
RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true);
Review Comment:
-> may operate same fs
What's the problem here, the fs instance is cached BTW, so you would still fetch the same fs instance each time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TJX2014 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
TJX2014 commented on code in PR #7105:
URL: https://github.com/apache/hudi/pull/7105#discussion_r1023882478
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java:
##########
@@ -79,7 +79,7 @@ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFi
result.addAll(context.flatMap(subDirectories, directory -> {
PathFilter pathFilter = getFilePathFilter(baseFileExtension);
Path path = new Path(directory);
- FileSystem fileSystem = path.getFileSystem(new Configuration());
+ FileSystem fileSystem = FSUtils.getFs(path, fs.getConf());
RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true);
Review Comment:
@danny0405 Hi, I have fixed it, please help review again ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1300707324
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315133318
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
}, {
"hash" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022",
"triggerID" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13024",
"triggerID" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"triggerType" : "PUSH"
}, {
"hash" : "2696db8089a9c8e1a5632378ae8f483a3a646d17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13029",
"triggerID" : "2696db8089a9c8e1a5632378ae8f483a3a646d17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ace488c3ed6ec5b91594a544932d5b4fb8ceddd5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13024)
* 2696db8089a9c8e1a5632378ae8f483a3a646d17 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1300176447
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84265e2e3f1c60d130fdd50be33e852f76386a0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706)
* 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1314944564
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
}, {
"hash" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022",
"triggerID" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728)
* 4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315126939
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
}, {
"hash" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022",
"triggerID" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13024",
"triggerID" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"triggerType" : "PUSH"
}, {
"hash" : "2696db8089a9c8e1a5632378ae8f483a3a646d17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2696db8089a9c8e1a5632378ae8f483a3a646d17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022)
* ace488c3ed6ec5b91594a544932d5b4fb8ceddd5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13024)
* 2696db8089a9c8e1a5632378ae8f483a3a646d17 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7105:
URL: https://github.com/apache/hudi/pull/7105#discussion_r1011083024
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java:
##########
@@ -79,7 +79,7 @@ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFi
result.addAll(context.flatMap(subDirectories, directory -> {
PathFilter pathFilter = getFilePathFilter(baseFileExtension);
Path path = new Path(directory);
- FileSystem fileSystem = path.getFileSystem(new Configuration());
+ FileSystem fileSystem = FSUtils.getFs(path, fs.getConf());
RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true);
Review Comment:
`fs` is already a `FileSystem`, why new an instance here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315036789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
}, {
"hash" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022",
"triggerID" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13024",
"triggerID" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022)
* ace488c3ed6ec5b91594a544932d5b4fb8ceddd5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13024)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-2346320479
Sounds good. Closing this stale PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-2345473265
I've assigned the JIRA ticket to myself. FileSystemBackedTableMetadata, FSUtils, and BootStrapUtils in this PR have already been refactored. I'll post a new PR to replace FS usages in DatePartitionPathSelector and related classes later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1298352785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84265e2e3f1c60d130fdd50be33e852f76386a0c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1298347820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84265e2e3f1c60d130fdd50be33e852f76386a0c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1300018399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84265e2e3f1c60d130fdd50be33e852f76386a0c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706)
* 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7105:
URL: https://github.com/apache/hudi/pull/7105#discussion_r1024725867
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapUtils.java:
##########
@@ -79,7 +79,7 @@ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFi
result.addAll(context.flatMap(subDirectories, directory -> {
PathFilter pathFilter = getFilePathFilter(baseFileExtension);
Path path = new Path(directory);
- FileSystem fileSystem = path.getFileSystem(new Configuration());
+ FileSystem fileSystem = FSUtils.getFs(path, fs.getConf());
RemoteIterator<LocatedFileStatus> itr = fileSystem.listFiles(path, true);
Review Comment:
What bug is fixed here then ? Or simplify a code refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1315417029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
}, {
"hash" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13022",
"triggerID" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"triggerType" : "PUSH"
}, {
"hash" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13024",
"triggerID" : "ace488c3ed6ec5b91594a544932d5b4fb8ceddd5",
"triggerType" : "PUSH"
}, {
"hash" : "2696db8089a9c8e1a5632378ae8f483a3a646d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13029",
"triggerID" : "2696db8089a9c8e1a5632378ae8f483a3a646d17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2696db8089a9c8e1a5632378ae8f483a3a646d17 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
URL: https://github.com/apache/hudi/pull/7105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] TJX2014 commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
TJX2014 commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1298307296
@danny0405 please help review this : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-2336864244
> @danny0405 is this still a bug or we can close the PR?
I kind of think it makes sense, we better to unify the instantiation of the fs objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-2344340890
Sounds good. @CTTY could you revamp this PR given you have worked on `HoodieStorage` related refactoring and improvement recently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7105: [HUDI-5128] Fix getFileSystem way in FileSystemBackedTableMetadata, DatePartitionPathSelector and BootstrapUtils not consistent issue
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7105:
URL: https://github.com/apache/hudi/pull/7105#issuecomment-1314937282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12706",
"triggerID" : "84265e2e3f1c60d130fdd50be33e852f76386a0c",
"triggerType" : "PUSH"
}, {
"hash" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728",
"triggerID" : "9c88f0cae5175e3159a2e5158bd43a0d21e5be55",
"triggerType" : "PUSH"
}, {
"hash" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9c88f0cae5175e3159a2e5158bd43a0d21e5be55 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12728)
* 4343f0f6e4372668d7e9619c09ce07fd0b1bcd7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org