[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6460: Core: Refactor ManifestListReadTask to avoid extra S3 calls

[GitBox <gi...@apache.org>]

szehon-ho commented on code in PR #6460:
URL: https://github.com/apache/iceberg/pull/6460#discussion_r1052809830


##########
core/src/main/java/org/apache/iceberg/AllManifestsTable.java:
##########
@@ -212,19 +200,22 @@ public CloseableIterable<StructLike> rows() {
         return CloseableIterable.transform(rowIterable, projection::wrap);
 
       } catch (IOException e) {
-        throw new RuntimeIOException(
-            e, "Cannot read manifest list file: %s", manifestListTask.file().path());
+        throw new RuntimeIOException(e, "Cannot read manifest list file: %s", manifestListLocation);
       }
     }
 
     @Override
     public DataFile file() {
-      return manifestListTask.file();
+      return DataFiles.builder(PartitionSpec.unpartitioned())

Review Comment:
   Optional: If this is called repeatedly maybe worth making this in the CTOR.



##########
core/src/main/java/org/apache/iceberg/AllManifestsTable.java:
##########
@@ -234,12 +225,14 @@ public long start() {
 
     @Override
     public long length() {
-      return manifestListTask.length();
+      // return a generic length to avoid looking up the actual length
+      return 8192;
     }
 
     @Override
     public Expression residual() {
-      return manifestListTask.residual();
+      // this table is unpartitioned so the residual is always constant
+      return residual;

Review Comment:
   Is this ever used?  I wonder can we just return ResidualEvaluator.unpartitioned(Expressions.alwaysFalse()) or something



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org