You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2026/08/21 00:43:46 UTC
[PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
rahil-c opened a new pull request, #19698:
URL: https://github.com/apache/hudi/pull/19698
### Describe the issue this Pull Request addresses
issue: #19697
Stacked on #19672, which should be reviewed and merged first. Until it merges this PR shows both commits.
Unstructured file ingestion can only discover files by listing a prefix, so every sync walks the whole prefix whether or not anything arrived. Hudi already avoids that for data files by reading a Hudi table of S3 or GCS notifications, but the step that turns the selected objects into rows was hardwired to a Spark datasource read, so documents could not use it.
### Summary and Changelog
Introduces `CloudObjectMaterializer`, the step that turns the objects a batch has selected into rows. Three concerns move together and so live on one type: the predicate deciding which object keys are eligible, the partition count, and the read itself. `ColumnarFileMaterializer` holds the previous behaviour and is registered as the default, and `generateFilter` keeps its existing two-argument form and delegates through it, so nothing changes for current pipelines.
`UnstructuredFileMaterializer` selects by document extension, sizes partitions by the bytes that will actually be parsed rather than the bytes referenced, and builds rows directly. An object above `parse.max.bytes` is referenced without being read and costs almost nothing, which makes a byte count of the whole batch a poor proxy for its cost. `UnstructuredFileS3EventsHoodieIncrSource` wires the two together. Row construction is now shared by both discovery paths through `UnstructuredFileRows`, so blob placement and parse handling exist once rather than once per source.
`CloudObjectMetadata` carries the notification timestamp, which the table needs to order repeated writes to the same path, and objects are de-duplicated on bucket and key keeping the newest event rather than on size as well: an object written twice inside one batch was previously read once per write. Where a metadata table predates the timestamp column the events cannot be ordered, so that falls back to the previous behaviour with a warning rather than failing.
### Impact
Additive. `ColumnarFileMaterializer` is the default and reproduces the previous read exactly, and no existing test file is modified by this PR, which is the evidence for that. The de-duplication change does affect existing cloud sources: an object rewritten inside a single batch is now read once instead of once per write. New user-facing config: `hoodie.streamer.source.unstructured.work.bytes.per.partition`.
### Risk Level
low. The behaviour-preserving claim is checked by the existing suites rather than asserted: `TestS3EventsHoodieIncrSource` (16), `TestGcsEventsHoodieIncrSource` (11), `TestCloudObjectsSelectorCommon` (29), `TestUnstructuredFileDFSSource` (2) and `TestCloudObjectsSelector` (50) all pass unchanged, including the argument captor that pins the partition count and the tests that pin the generated filter string.
Validated end to end against real S3 on a 4,032 object corpus spread over 1,008 prefixes: the new source produces a table identical in content to the folder-based source, 4,044 rows with 4,044 successful parses after 12 further objects were added. A discovery-cost comparison on the same corpus measured 121.8s by listing against 12.3s by notification for a sync with nothing to ingest.
Two defects were found by that run rather than by the suite, and both are fixed here with tests that reproduce them: the de-duplication ranked over nested columns after projecting them, which renames them to their leaf names and fails the batch with `UNRESOLVED_COLUMN`; and the rendered object-key predicate was built from a hash set, so its order was arbitrary across runs.
### Documentation Update
New config documented on the config class, so the reference tables generate from it. The website page describing unstructured ingestion will need the new source class name and a note that `hoodie.streamer.source.s3incr.fs.prefix` must match the registered filesystem scheme, since `s3` is the default while `s3a` is what `hadoop-aws` registers.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19698:
URL: https://github.com/apache/hudi/pull/19698#discussion_r3839460516
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -212,7 +216,26 @@ private static Option<CloudObjectMetadata> processRow(Row row, String storageUrl
} else {
throw new HoodieIOException("unexpected object size's type in Cloud storage events: " + obj.getClass());
}
- return Option.of(new CloudObjectMetadata(url, size));
+ long modificationTime = row.size() > 3
+ ? epochMillis(row.get(3)) : CloudObjectMetadata.UNKNOWN_MODIFICATION_TIME;
+ return Option.of(new CloudObjectMetadata(url, size, modificationTime));
+ }
+
+ /**
+ * Epoch millis for a notification timestamp, which both S3 and GCS report as an ISO-8601 string.
+ * Anything unparseable yields {@link CloudObjectMetadata#UNKNOWN_MODIFICATION_TIME} rather than
+ * failing the batch, since the value only orders writes to the same object.
+ */
+ private static long epochMillis(Object rawValue) {
+ if (rawValue == null) {
+ return CloudObjectMetadata.UNKNOWN_MODIFICATION_TIME;
+ }
+ try {
+ return Instant.parse(rawValue.toString()).toEpochMilli();
+ } catch (DateTimeParseException e) {
+ log.warn("Ignoring unparseable cloud notification timestamp {}", rawValue);
Review Comment:
`processRow` runs inside the `mapPartitions` function, so this warns once per row on the executors. A batch whose timestamps are all in an unexpected format would log a line per object.
The reason I would care beyond the noise: every UNKNOWN sends `buildRow(fs, object)` down the `buildRow(fs, path)` fallback, which stats the object. So a format mismatch silently converts the notification path back into one `getFileStatus` per object, which is most of the 12.3s versus 121.8s advantage you measured. Worth logging once per batch and counting, so the degradation is visible?
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -362,6 +416,33 @@ public static List<CloudObjectMetadata> getObjectMetadata(
.collectAsList();
}
+ /**
+ * One row per cloud object, carrying the notification timestamp when the events have one.
+ *
+ * <p>An object written more than once inside a single batch produces one event per write. Keying
+ * only on bucket and object key, and keeping the newest event, reads such an object once instead
+ * of once per write. Where the metadata table predates the timestamp column the events cannot be
+ * ordered, so this falls back to the previous behaviour of de-duplicating on size as well.
+ */
+ private static Dataset<Row> selectDistinctObjects(Dataset<Row> events, String bucketCol, String keyCol,
+ String sizeCol, String timeCol) {
+ if (!Arrays.asList(events.schema().fieldNames()).contains(timeCol)) {
+ log.warn("Cloud events carry no {} column; objects rewritten within a batch will be read once per write", timeCol);
+ return events.select(bucketCol, keyCol, sizeCol).distinct();
+ }
+ String rank = "_hoodie_event_rank";
+ // rank before projecting: the columns are nested, so selecting them first renames them to
+ // their leaf names and the window would no longer resolve bucketCol or keyCol.
+ // Order on the parsed instant, not the raw string: '.' sorts before 'Z', so a lexicographic
+ // desc puts 10:00:00Z ahead of the later 10:00:00.500Z whenever precision varies in a second.
+ return events
+ .withColumn(rank, functions.row_number().over(
+ Window.partitionBy(functions.col(bucketCol), functions.col(keyCol))
+ .orderBy(functions.to_timestamp(functions.col(timeCol)).desc_nulls_last())))
Review Comment:
Fixed in aa8f9a0, flagging it here for the record since it is a correctness change rather than a refactor. Ranking on the raw string made the order lexicographic: a same-second pair at `10:00:00Z` and `10:00:00.500Z` picked the earlier write, because `'.'` sorts before `'Z'`. Because `modification_time` is the precombine field, the true newer event then loses as stale on a later batch and the table keeps the superseded content.
One loose end I deliberately left out of that commit: `to_timestamp` yields null on anything it cannot parse, and those rows tie under `desc_nulls_last`, so `row_number` still picks arbitrarily among them. A tiebreak on the key or size would close that if you think it is worth it.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudDataFetcher.java:
##########
@@ -144,15 +152,9 @@ private Option<Dataset<Row>> getCloudObjectDataDF(List<CloudObjectMetadata> clou
for (CloudObjectMetadata o : cloudObjectMetadata) {
totalSize += o.getSize();
Review Comment:
nit: `totalSize` is summed here for the metric and again inside `ColumnarFileMaterializer.partitionCount`. Also, for the unstructured materializer this metric now reports bytes *referenced*, which that class's own javadoc argues is a poor proxy for the batch's cost, so the number means something different depending on which materializer is in play.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -212,7 +216,26 @@ private static Option<CloudObjectMetadata> processRow(Row row, String storageUrl
} else {
throw new HoodieIOException("unexpected object size's type in Cloud storage events: " + obj.getClass());
}
- return Option.of(new CloudObjectMetadata(url, size));
+ long modificationTime = row.size() > 3
Review Comment:
Concrete way this reads UNKNOWN for a whole batch: if `eventTime` is stored as a Spark `TimestampType` rather than a string, `row.get(3).toString()` is `2026-08-12 10:00:00.5`, which `Instant.parse` rejects. Also worth noting that Spark's `to_timestamp` in the ranking window and `Instant.parse` here are two different parsers, so a value Spark accepts can still land as UNKNOWN.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/UnstructuredFileMaterializer.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+import org.apache.hudi.utilities.sources.helpers.unstructured.UnstructuredFileRecordBuilder;
+import org.apache.hudi.utilities.sources.helpers.unstructured.UnstructuredFileRows;
+
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import static org.apache.hudi.common.util.ConfigUtils.getIntWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+import static org.apache.hudi.utilities.config.UnstructuredFileSourceConfig.FILE_EXTENSIONS;
+import static org.apache.hudi.utilities.config.UnstructuredFileSourceConfig.FILE_EXTENSIONS_IGNORE;
+import static org.apache.hudi.utilities.config.UnstructuredFileSourceConfig.LISTING_PARALLELISM;
+import static org.apache.hudi.utilities.config.UnstructuredFileSourceConfig.PARSE_MAX_BYTES;
+import static org.apache.hudi.utilities.config.UnstructuredFileSourceConfig.WORK_BYTES_PER_PARTITION;
+
+/**
+ * Reads cloud objects as unstructured files: each object becomes one row carrying a BLOB column
+ * plus extracted text, metadata and chunks, the same shape the directory-listing source produces.
+ *
+ * <p>Differs from a columnar read in all three of its responsibilities. Objects are selected by
+ * document extension rather than by data-file format. Partitions are sized by the bytes that will
+ * actually be parsed, because an object above {@code parse.max.bytes} is referenced without being
+ * read and so costs almost nothing. And rows are built directly rather than through a Spark
+ * datasource.
+ */
+public class UnstructuredFileMaterializer implements CloudObjectMaterializer {
+
+ private static final long serialVersionUID = 1L;
+
+ private final Set<String> allowedExtensions;
+ private final Set<String> ignoredExtensions;
+ private final long parseMaxBytes;
+ private final long workBytesPerPartition;
+ private final int configuredParallelism;
+ private final int defaultParallelism;
+ private final UnstructuredFileRecordBuilder recordBuilder;
+
+ public UnstructuredFileMaterializer(TypedProperties props, JavaSparkContext jsc) {
+ this.allowedExtensions = UnstructuredFileRows.parseExtensions(getStringWithAltKeys(props, FILE_EXTENSIONS, true));
+ this.ignoredExtensions = UnstructuredFileRows.parseExtensions(getStringWithAltKeys(props, FILE_EXTENSIONS_IGNORE, true));
+ this.parseMaxBytes = getLongWithAltKeys(props, PARSE_MAX_BYTES);
+ this.workBytesPerPartition = getLongWithAltKeys(props, WORK_BYTES_PER_PARTITION);
+ this.configuredParallelism = getIntWithAltKeys(props, LISTING_PARALLELISM);
+ this.defaultParallelism = jsc.defaultParallelism();
+ this.recordBuilder = new UnstructuredFileRecordBuilder(props);
+ }
+
+ @Override
+ public String objectKeyPredicate(String objectKey, TypedProperties props) {
+ if (!allowedExtensions.isEmpty()) {
+ return CloudObjectsSelectorCommon.extensionPredicate(objectKey, String.join(",", allowedExtensions));
+ }
+ if (ignoredExtensions.isEmpty()) {
+ return "";
+ }
+ // no allowlist, so select everything except the denied extensions
+ List<String> denials = new ArrayList<>();
+ for (String extension : ignoredExtensions) {
+ denials.add(String.format("%s not like '%%%s'", objectKey, extension));
+ }
+ return " and " + String.join(" and ", denials);
+ }
+
+ @Override
+ public int partitionCount(List<CloudObjectMetadata> objects, long bytesPerPartition, int minPartitions) {
Review Comment:
This is the interesting half of the class and codecov has it at 0%. The parseable-bytes sum, the `configuredParallelism`/`defaultParallelism` floor and the `minPartitions` max are all worth pinning, especially since the whole point is that it deliberately differs from the columnar sizing. Could you add a unit test for it, and one for the denylist branch of `objectKeyPredicate`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5363898902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433",
"triggerID" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 039aa8b104c6d86e9837a829ed3de5bd0b62b27b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19698:
URL: https://github.com/apache/hudi/pull/19698#discussion_r3839312786
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectMaterializer.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Turns the cloud objects an incremental source has selected into rows. Selecting which objects
+ * to read is common to every payload, so it stays in {@link CloudDataFetcher}; how those objects
+ * become rows is not, which is what implementations supply.
+ *
+ * <p>Three concerns move together and so belong on one type: the predicate that decides which
+ * object keys are eligible, how many Spark partitions the batch is spread over, and the read
+ * itself. A columnar reader selects by data-file format, sizes partitions by bytes scanned and
+ * defers to a Spark datasource; a reader that parses documents selects by document extension,
+ * sizes partitions by the bytes it will actually parse, and builds rows itself.
+ */
+public interface CloudObjectMaterializer extends Serializable {
Review Comment:
I would like to us go deep on the need for the following interface? Prior to the interface how did objects get materailzied? Lets think carefully on what the right abstractions are here around handling structed and unstructured data.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectMaterializer.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Turns the cloud objects an incremental source has selected into rows. Selecting which objects
+ * to read is common to every payload, so it stays in {@link CloudDataFetcher}; how those objects
+ * become rows is not, which is what implementations supply.
+ *
+ * <p>Three concerns move together and so belong on one type: the predicate that decides which
+ * object keys are eligible, how many Spark partitions the batch is spread over, and the read
+ * itself. A columnar reader selects by data-file format, sizes partitions by bytes scanned and
+ * defers to a Spark datasource; a reader that parses documents selects by document extension,
+ * sizes partitions by the bytes it will actually parse, and builds rows itself.
+ */
+public interface CloudObjectMaterializer extends Serializable {
Review Comment:
I would like to us go deep on the need for the following interface? Prior to the interface how did objects get materailzied? Lets think carefully on what the right abstractions are here around handling structured and unstructured data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5363902106
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19698?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `81.81818%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 77.82%. Comparing base ([`5a11a4f`](https://app.codecov.io/gh/apache/hudi/commit/5a11a4fce20620afac143358bc771dfe2d14d3c8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`039aa8b`](https://app.codecov.io/gh/apache/hudi/commit/039aa8b104c6d86e9837a829ed3de5bd0b62b27b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19698?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...es/sources/helpers/CloudObjectsSelectorCommon.java](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FCloudObjectsSelectorCommon.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9DbG91ZE9iamVjdHNTZWxlY3RvckNvbW1vbi5qYXZh) | 80.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19698 +/- ##
============================================
- Coverage 77.83% 77.82% -0.01%
+ Complexity 33140 33120 -20
============================================
Files 2527 2527
Lines 139942 139881 -61
Branches 16849 16847 -2
============================================
- Hits 108926 108865 -61
- Misses 23396 23401 +5
+ Partials 7620 7615 -5
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.34% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.82% <ø> (ø)` | |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.71% <ø> (-0.04%)` | :arrow_down: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.28% <ø> (+<0.01%)` | :arrow_up: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.96% <81.81%> (-0.08%)` | :arrow_down: |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.09% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.55% <ø> (-0.03%)` | :arrow_down: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.85% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.33% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19698/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19698/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19698/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.90% <ø> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19698?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...pache/hudi/utilities/config/CloudSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FCloudSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9DbG91ZFNvdXJjZUNvbmZpZy5qYXZh) | `99.27% <100.00%> (ø)` | |
| [...utilities/config/UnstructuredFileSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FUnstructuredFileSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9VbnN0cnVjdHVyZWRGaWxlU291cmNlQ29uZmlnLmphdmE=) | `98.41% <ø> (ø)` | |
| [...i/utilities/sources/UnstructuredFileDFSSource.java](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2FUnstructuredFileDFSSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvVW5zdHJ1Y3R1cmVkRmlsZURGU1NvdXJjZS5qYXZh) | `92.30% <ø> (+4.80%)` | :arrow_up: |
| [...di/utilities/sources/helpers/CloudDataFetcher.java](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FCloudDataFetcher.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9DbG91ZERhdGFGZXRjaGVyLmphdmE=) | `94.11% <ø> (-0.62%)` | :arrow_down: |
| [...rs/unstructured/UnstructuredFileRecordBuilder.java](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2Funstructured%2FUnstructuredFileRecordBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy91bnN0cnVjdHVyZWQvVW5zdHJ1Y3R1cmVkRmlsZVJlY29yZEJ1aWxkZXIuamF2YQ==) | `83.67% <ø> (-0.65%)` | :arrow_down: |
| [...es/sources/helpers/CloudObjectsSelectorCommon.java](https://app.codecov.io/gh/apache/hudi/pull/19698?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FCloudObjectsSelectorCommon.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9DbG91ZE9iamVjdHNTZWxlY3RvckNvbW1vbi5qYXZh) | `78.78% <80.00%> (-0.92%)` | :arrow_down: |
... and [12 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19698/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5388208030
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433",
"triggerID" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"triggerType" : "PUSH"
}, {
"hash" : "aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16492",
"triggerID" : "aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 039aa8b104c6d86e9837a829ed3de5bd0b62b27b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433)
* aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16492)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5363862580
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 039aa8b104c6d86e9837a829ed3de5bd0b62b27b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5364895180
Self-review pass on this stack before it comes out of draft. Nothing blocking; the materializer seam looks like the right abstraction. Five things worth a second look, most significant first.
**1. `selectDistinctObjects` changes behaviour for every existing cloud source, not just the unstructured one** (`CloudObjectsSelectorCommon.java`, `selectDistinctObjects`)
This used to be `select(bucket, key, size).distinct()`. It now keeps only the newest event per `(bucket, key)` when the events carry a timestamp, so an object written twice inside one batch with different sizes is read once instead of twice. That looks like a genuine fix, but it lands in a PR about unstructured ingestion and it adds a `Window` shuffle to the hottest path in cloud ingestion. Worth calling out explicitly in the description, or splitting into its own commit, so a reviewer of the structured path sees it.
**2. An unparseable notification timestamp degrades silently and noisily, per object** (`CloudObjectsSelectorCommon.epochMillis`)
`epochMillis` runs per row inside `mapPartitions`, so each unparseable value logs a WARN. It then returns `UNKNOWN_MODIFICATION_TIME`, which makes `buildRow` fall back to `fs.getFileStatus` - turning the "avoid a metadata request per object" optimisation into exactly one metadata request per object. A 100k-object batch whose timestamp column is typed as Spark `TimestampType` (whose `toString` is `2026-08-20 10:00:00.0`, not ISO-8601) would produce 100k warnings and 100k HEAD requests, with nothing in the logs explaining the slowdown. Could this be detected once per batch and logged once?
**3. Partition count is sized by bytes, but work is distributed by object count** (`UnstructuredFileRows.toDataset`)
`UnstructuredFileMaterializer.partitionCount` computes the partition count from parseable bytes, then `jsc.parallelize(objects, partitions)` splits the list by element count. A batch of one 100 MiB document plus 10,000 small files can land the large document and its share of small ones in a single task. `CloudObjectMetadata` now carries `size`, so bucketing into partitions by bytes is nearly free here.
**4. `partitionCount` and `toDataset` are untested** (`UnstructuredFileMaterializer`, `UnstructuredFileRows`)
The predicate and dedup paths are covered, which is the right instinct. The gap is `partitionCount`, whose `Math.max(byWork, Math.min(floor, Math.max(objects.size(), 1)))` silently decides parallelism - a regression there surfaces only as a slow or skewed job. Two cases would cover it: every object above `parse.max.bytes`, and a batch smaller than `defaultParallelism`.
**5. Is the materializer ever serialized?** (`ColumnarFileMaterializer`)
`selectorCommon` is `transient` on a `Serializable` type, held by a `CloudDataFetcher` that declares a `serialVersionUID`. Driver-only today, so this is latent rather than live, but if the fetcher ever travels to an executor `materialize()` would NPE. A line on the class stating the driver-only contract would make that safe to rely on.
**Candidates I checked and discarded**, recorded so nobody re-raises them:
- The extension-filter default is unchanged: `ColumnarFileMaterializer.objectKeyPredicate` keeps `.orElse(fileFormat)`, and `extensionPredicate` renders a byte-identical string for the single-extension case.
- `modification_time` is the precombine field, so a `0` from a notification would be a real hazard - but `buildRow(fs, CloudObjectMetadata)` falls back to `fs.getFileStatus` on `UNKNOWN_MODIFICATION_TIME`. Handled deliberately.
- The positional `row.get(3)` for the timestamp holds on all three paths: with the exists check (`repartition` preserves columns), without it, and when the metadata table has no timestamp column (three columns, caught by the `row.size() > 3` guard).
- `partitionCount` ignoring bytes that are inlined but not parsed is only reachable if `parse.max.bytes` is configured below `blob.inline.max.bytes` (defaults are 128 MiB and 1 MiB), and the `floor` holds parallelism at `defaultParallelism` regardless. Rows are lazy, so peak task memory stays at one blob.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5388181447
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433",
"triggerID" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"triggerType" : "PUSH"
}, {
"hash" : "aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 039aa8b104c6d86e9837a829ed3de5bd0b62b27b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433)
* aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5388449058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433",
"triggerID" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"triggerType" : "PUSH"
}, {
"hash" : "aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16492",
"triggerID" : "aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aa8f9a0980e6cf7e93e6b01a1159e53533cdacd1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16492)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19698:
URL: https://github.com/apache/hudi/pull/19698#issuecomment-5364232355
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433",
"triggerID" : "039aa8b104c6d86e9837a829ed3de5bd0b62b27b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 039aa8b104c6d86e9837a829ed3de5bd0b62b27b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16433)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): ingest unstructured files announced by S3 event notifications [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19698:
URL: https://github.com/apache/hudi/pull/19698#discussion_r3839332056
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/ColumnarFileMaterializer.java:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.sources.helpers;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.utilities.schema.SchemaProvider;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.SparkSession;
+
+import java.util.List;
+
+import static org.apache.hudi.utilities.config.CloudSourceConfig.CLOUD_DATAFILE_EXTENSION;
+
+/**
+ * Reads cloud objects as data files through a Spark datasource, which is what every cloud
+ * incremental source did before materializers existed. Selects by the configured data file
+ * extension, defaulting to the data file format, and sizes partitions by the bytes referenced.
+ * This is the default materializer, so leaving it unset preserves existing behaviour.
+ */
+public class ColumnarFileMaterializer implements CloudObjectMaterializer {
Review Comment:
Is the behavior of this class the same 1-1 logic prior to this refactoring?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org