You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/09/26 18:23:56 UTC
[GitHub] [pinot] klsince opened a new pull request, #9466: collect file info like mtime, length while listing files for free
klsince opened a new pull request, #9466:
URL: https://github.com/apache/pinot/pull/9466
This PR extends the PinotFS interface a bit so that one can get file info like mtime and length for free while calling listFiles(), otherwise one has to call length(), lastModificationTime() to get the info with an extra round trip, which can be costly when the input folder is large.
Plan to get quick feedbacks with this PR, and extend the other implementations in following PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r980392494
##########
pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/FileInfo.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.filesystem;
+
+import com.google.common.base.Preconditions;
+import java.util.Optional;
+import org.apache.commons.lang3.StringUtils;
+
+
+/**
+ * FileInfo contains the file path and many optional file attributes like mtime, length etc.
+ */
+public class FileInfo {
+ private final String _filePath;
+ private final Long _lastModifiedTime;
Review Comment:
I thought some future impl might not have all those attributes ready during file listing, so I've made them Optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r1028584776
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
+ // Only add files and not directories
Review Comment:
I think the question was about recursive listing, which actually works as expected.
The comment may be misleading. But it means that the returned list of Strings are just file paths, w/o subfolders' paths; it doesn't mean subfolders are skipped during recursive listing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9466:
URL: https://github.com/apache/pinot/pull/9466#issuecomment-1258819331
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9466?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9466](https://codecov.io/gh/apache/pinot/pull/9466?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cfb0093) into [master](https://codecov.io/gh/apache/pinot/commit/e2d8aaab7236aa8dccf6b5cfaefd3d98ea0e4639?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e2d8aaa) will **decrease** coverage by `2.70%`.
> The diff coverage is `89.65%`.
```diff
@@ Coverage Diff @@
## master #9466 +/- ##
============================================
- Coverage 69.83% 67.13% -2.71%
+ Complexity 5127 4939 -188
============================================
Files 1909 1414 -495
Lines 101749 73862 -27887
Branches 15440 11800 -3640
============================================
- Hits 71059 49588 -21471
+ Misses 25674 20683 -4991
+ Partials 5016 3591 -1425
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `67.13% <89.65%> (+0.01%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../java/org/apache/pinot/spi/filesystem/PinotFS.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZmlsZXN5c3RlbS9QaW5vdEZTLmphdmE=) | `50.00% <0.00%> (-10.00%)` | :arrow_down: |
| [...java/org/apache/pinot/spi/filesystem/FileInfo.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZmlsZXN5c3RlbS9GaWxlSW5mby5qYXZh) | `90.47% <90.47%> (ø)` | |
| [.../org/apache/pinot/spi/filesystem/LocalPinotFS.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZmlsZXN5c3RlbS9Mb2NhbFBpbm90RlMuamF2YQ==) | `82.25% <100.00%> (+2.25%)` | :arrow_up: |
| [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/common/config/NettyConfig.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL05ldHR5Q29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ache/pinot/server/access/AccessControlFactory.java](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL2FjY2Vzcy9BY2Nlc3NDb250cm9sRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [742 more](https://codecov.io/gh/apache/pinot/pull/9466/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r1028532956
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
+ // Only add files and not directories
Review Comment:
Can you elaborate more on how it will break?
In S3, there is no folder concept under a bucket (flat structure), and that's probably the reason why folder is not returned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
klsince commented on PR #9466:
URL: https://github.com/apache/pinot/pull/9466#issuecomment-1258709323
> ... we should implement it for all PinotFS
Included S3 and Local in this PR, and plan to extend the others in next PR. Lemme know if you want to have all of them here. I just wanted to iterate in small steps faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r1028741045
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
+ // Only add files and not directories
Review Comment:
yeah. even though S3 is an object store it does provide the file-system based APIs. Users might not know the differences since they are all PinotFS impl to users and thus it is reasonable to expect same behavior
i think for here, either we dont allow recursive on S3 (explicitly throw exception if that config is given); or we can do the recursive properly (i dont know how or even if that's possible though)
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
+ // Only add files and not directories
Review Comment:
yeah. even though S3 is an object store it does provide the file-system like APIs. Users might not know the differences since they are all PinotFS impl to users and thus it is reasonable to expect same behavior
i think for here, either we dont allow recursive on S3 (explicitly throw exception if that config is given); or we can do the recursive properly (i dont know how or even if that's possible though)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r1021791705
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
+ // Only add files and not directories
Review Comment:
seems like this would break if we enable recursive file ingestion on the ingestion task config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r1028780092
##########
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java:
##########
@@ -435,8 +437,48 @@ public long length(URI fileUri)
@Override
public String[] listFiles(URI fileUri, boolean recursive)
throws IOException {
+ ImmutableList.Builder<String> builder = ImmutableList.builder();
+ visitFiles(fileUri, recursive, s3Object -> {
+ // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
+ // Only add files and not directories
Review Comment:
Recursive listing is implemented in visitFiles() method below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #9466:
URL: https://github.com/apache/pinot/pull/9466
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r980386228
##########
pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/FileInfo.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.filesystem;
+
+import com.google.common.base.Preconditions;
+import java.util.Optional;
+import org.apache.commons.lang3.StringUtils;
+
+
+/**
+ * FileInfo contains the file path and many optional file attributes like mtime, length etc.
+ */
+public class FileInfo {
+ private final String _filePath;
+ private final Long _lastModifiedTime;
Review Comment:
Suggest making `_lastModifiedTime`, `_length` and `_isDirectory` primitive type to be consistent with other APIs. Essentially it should return the same info as calling each individual API
##########
pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/FileInfo.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.filesystem;
+
+import com.google.common.base.Preconditions;
+import java.util.Optional;
+import org.apache.commons.lang3.StringUtils;
+
+
+/**
+ * FileInfo contains the file path and many optional file attributes like mtime, length etc.
+ */
+public class FileInfo {
+ private final String _filePath;
+ private final Long _lastModifiedTime;
+ private final Long _length;
+ private final Boolean _isDirectory;
+
+ private FileInfo(String filePath, Long lastModifiedTime, Long length, Boolean isDirectory) {
+ _filePath = filePath;
+ _lastModifiedTime = lastModifiedTime;
+ _length = length;
+ _isDirectory = isDirectory;
+ }
+
+ public String getFilePath() {
+ return _filePath;
+ }
+
+ public Optional<Long> getLastModifiedTime() {
+ return Optional.ofNullable(_lastModifiedTime);
+ }
+
+ public Optional<Long> getLength() {
+ return Optional.ofNullable(_length);
+ }
+
+ public Optional<Boolean> isDirectory() {
+ return Optional.ofNullable(_isDirectory);
+ }
+
+ @Override
+ public String toString() {
+ return "FileInfo{" + "_filePath='" + _filePath + '\'' + ", _lastModifiedTime=" + _lastModifiedTime + ", _length="
+ + _length + ", _isDirectory=" + _isDirectory + '}';
+ }
+
+ public static class Builder {
+ private String _filePath;
+ private Long _lastModifiedTime;
+ private Long _length;
+ private Boolean _isDirectory;
+
+ public Builder setFilePath(String filePath) {
+ _filePath = filePath;
+ return this;
+ }
+
+ public Builder setLastModifiedTime(long lastModifiedTime) {
+ _lastModifiedTime = lastModifiedTime;
+ return this;
+ }
+
+ public Builder setLength(long length) {
+ _length = length;
+ return this;
+ }
+
+ public Builder setIsDirectory(boolean isDirectory) {
+ _isDirectory = isDirectory;
+ return this;
+ }
+
+ public FileInfo build() {
+ Preconditions.checkArgument(StringUtils.isNotEmpty(_filePath), "The filePath is required");
+ return new FileInfo(_filePath, _lastModifiedTime, _length, _isDirectory);
+ }
+ }
+}
Review Comment:
(format) add empty line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9466: collect file info like mtime, length while listing files for free
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9466:
URL: https://github.com/apache/pinot/pull/9466#discussion_r980457361
##########
pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/FileInfo.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.spi.filesystem;
+
+import com.google.common.base.Preconditions;
+import java.util.Optional;
+import org.apache.commons.lang3.StringUtils;
+
+
+/**
+ * FileInfo contains the file path and many optional file attributes like mtime, length etc.
+ */
+public class FileInfo {
+ private final String _filePath;
+ private final Long _lastModifiedTime;
Review Comment:
on a second thought, since there is the `String[] listFiles()` method, if some future impl can't fill those attributes during listing, then they should just just throw `Unsupported...Exception` and let callers use the basic listFiles()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org