[GitHub] [hudi] codope commented on a diff in pull request #8290: [HUDI-5983] Improve loading data via cloud store incr source

["codope (via GitHub)" <gi...@apache.org>]

codope commented on code in PR #8290:
URL: https://github.com/apache/hudi/pull/8290#discussion_r1148822988


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -115,4 +129,41 @@ private static boolean checkIfFileExists(String storageUrlSchemePrefix, String b
       throw new HoodieIOException(errMsg, ioe);
     }
   }
+
+  public static Option<Dataset<Row>> loadAsDataset(SparkSession spark, List<CloudObject> cloudObjects, TypedProperties props, String fileFormat) {
+    LOG.debug("Extracted distinct files " + cloudObjects.size()
+        + " and some samples " + cloudObjects.stream().map(CloudObject::getPath).limit(10).collect(Collectors.toList()));
+
+    if (isNullOrEmpty(cloudObjects)) {
+      return Option.empty();
+    }
+    DataFrameReader reader = spark.read().format(fileFormat);
+    String datasourceOpts = props.getString(SPARK_DATASOURCE_OPTIONS, null);
+    if (StringUtils.isNullOrEmpty(datasourceOpts)) {
+      // fall back to legacy config for BWC. TODO consolidate in HUDI-5780
+      datasourceOpts = props.getString(S3EventsHoodieIncrSource.Config.SPARK_DATASOURCE_OPTIONS, null);
+    }
+    if (StringUtils.nonEmpty(datasourceOpts)) {
+      final ObjectMapper mapper = new ObjectMapper();
+      Map<String, String> sparkOptionsMap = null;
+      try {
+        sparkOptionsMap = mapper.readValue(datasourceOpts, Map.class);
+      } catch (IOException e) {
+        throw new HoodieException(String.format("Failed to parse sparkOptions: %s", datasourceOpts), e);
+      }
+      LOG.info(String.format("sparkOptions loaded: %s", sparkOptionsMap));
+      reader = reader.options(sparkOptionsMap);
+    }
+    List<String> paths = new ArrayList<>();
+    long totalSize = 0;
+    for (CloudObject o: cloudObjects) {
+      paths.add(o.getPath());
+      totalSize += o.getSize();
+    }
+    // inflate 10% for potential hoodie meta fields
+    totalSize *= 1.1;

Review Comment:
   Should we be more conservative? Is 10% enough?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org