You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2026/08/19 22:39:01 UTC
[PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
rahil-c opened a new pull request, #19672:
URL: https://github.com/apache/hudi/pull/19672
### Describe the issue this Pull Request addresses
closes #19671
`hoodie.streamer.source.cloud.data.select.file.extension` is interpolated into a single SQL predicate, so a bucket holding more than one file type cannot be ingested by one streamer. Setting a list looks like it should work and fails quietly: `select.file.extension=json,jsonl` generates `s3.object.key like '%json,jsonl'`, a literal match on the joined string that selects nothing, and the job then commits nothing without reporting an error.
### Summary and Changelog
A comma separated value now renders one `like` predicate per extension, OR'd inside parentheses so it composes correctly with the size and relative-path predicates. A single value renders exactly the predicate it did before, and the fallback to the data file format when the config is unset is unchanged. `generateFilter` had no test coverage, so six cases were added to the existing suite: single and multiple extensions, blank and empty list entries, the format fallback, the GCS column names, and composition with the path prefix filter.
### Impact
Additive and backwards compatible. Existing single-extension pipelines produce a byte-identical filter, and the config keeps its current default. Users with mixed-extension buckets can now select them in one pipeline instead of running one per extension.
### Risk Level
low. The change is confined to rendering one config value into a filter string, and the previously untested single-value and format-fallback behaviours are now pinned by tests. Verified by running the new cases against the unmodified code first, where the four multiple-extension cases fail as expected (`expected: ...like '%pdf' or ...like '%docx'` but `was: ...like '%pdf,docx'`) while the two backwards-compatibility cases pass both before and after. `TestCloudObjectsSelectorCommon` (27), `TestS3EventsHoodieIncrSource` (16), `TestGcsEventsHoodieIncrSource` (11), `TestCloudObjectsSelector` (50) and `TestDeprecatedCloudIngestionConfigs` (3) all pass, with checkstyle clean.
### Documentation Update
The config documentation on `CLOUD_DATAFILE_EXTENSION` is updated in this PR and the reference tables generate from it, so no separate website change is needed.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #19672: feat(utilities): select more than one file extension in cloud incremental sources
URL: https://github.com/apache/hudi/pull/19672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5348991369
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19672?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 77.83%. Comparing base ([`5a11a4f`](https://app.codecov.io/gh/apache/hudi/commit/5a11a4fce20620afac143358bc771dfe2d14d3c8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8241539`](https://app.codecov.io/gh/apache/hudi/commit/82415395df915019895bb38200cd0753a50aee30?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19672 +/- ##
============================================
- Coverage 77.83% 77.83% -0.01%
+ Complexity 33140 33139 -1
============================================
Files 2527 2527
Lines 139942 139940 -2
Branches 16849 16849
============================================
- Hits 108926 108922 -4
- Misses 23396 23397 +1
- Partials 7620 7621 +1
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.34% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.82% <ø> (-0.01%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.75% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.27% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.03% <ø> (-0.01%)` | :arrow_down: |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.09% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.58% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.85% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.33% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19672/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19672/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19672/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.90% <ø> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19672?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...pache/hudi/utilities/config/CloudSourceConfig.java](https://app.codecov.io/gh/apache/hudi/pull/19672?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fconfig%2FCloudSourceConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2NvbmZpZy9DbG91ZFNvdXJjZUNvbmZpZy5qYXZh) | `99.27% <ø> (-0.01%)` | :arrow_down: |
| [...es/sources/helpers/CloudObjectsSelectorCommon.java](https://app.codecov.io/gh/apache/hudi/pull/19672?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fsources%2Fhelpers%2FCloudObjectsSelectorCommon.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvaGVscGVycy9DbG91ZE9iamVjdHNTZWxlY3RvckNvbW1vbi5qYXZh) | `79.62% <ø> (-0.08%)` | :arrow_down: |
... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19672/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5376631652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82415395df915019895bb38200cd0753a50aee30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392",
"triggerID" : "82415395df915019895bb38200cd0753a50aee30",
"triggerType" : "PUSH"
}, {
"hash" : "4f57edf018c37497f88731e2bd906c5ee8fa1f72",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16475",
"triggerID" : "4f57edf018c37497f88731e2bd906c5ee8fa1f72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82415395df915019895bb38200cd0753a50aee30 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392)
* 4f57edf018c37497f88731e2bd906c5ee8fa1f72 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16475)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5348924517
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82415395df915019895bb38200cd0753a50aee30",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "82415395df915019895bb38200cd0753a50aee30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82415395df915019895bb38200cd0753a50aee30 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19672:
URL: https://github.com/apache/hudi/pull/19672#discussion_r3834695475
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/CloudSourceConfig.java:
##########
@@ -131,7 +131,9 @@ public class CloudSourceConfig extends HoodieConfig {
.noDefaultValue()
.withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "source.cloud.data.select.file.extension")
.markAdvanced()
- .withDocumentation("Only match files with this extension. By default, this is the same as hoodie.streamer.source.hoodieincr.file.format");
+ .withDocumentation("Only match files with this extension. Accepts a comma separated list, "
Review Comment:
🤖 Traced it — the doc is close but slightly imprecise. When `CLOUD_DATAFILE_EXTENSION` is unset, `CloudObjectsSelectorCommon` falls back to `fileFormat = CloudDataFetcher.getFileFormat(props)` (line 306). `getFileFormat` returns `DATAFILE_FORMAT` (`...cloud.data.datafile.format`) when that's set, and only falls back to `SOURCE_FILE_FORMAT` (`hoodie.streamer.source.hoodieincr.file.format`, default `parquet`) when it isn't. So the extension default really tracks the effective data file format, which equals `hoodieincr.file.format` only when `cloud.data.datafile.format` is also unset. Might be worth wording the doc as "defaults to the resolved data file format" to capture that.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java:
##########
@@ -302,13 +302,33 @@ public static String generateFilter(Type type,
filter.append(SPACE_DELIMTER).append(String.format("and %s not like '%%%s%%'", objectKey, ignoreRelativePathSubStr.get()));
}
- // Match files with a given extension, or use the fileFormat as the default.
+ // Match files with any of the given extensions, or use the fileFormat as the default.
getPropVal(props, CLOUD_DATAFILE_EXTENSION).or(() -> Option.of(fileFormat))
- .map(val -> filter.append(SPACE_DELIMTER).append(String.format("and %s like '%%%s'", objectKey, val)));
+ .map(extensions -> filter.append(extensionClause(objectKey, extensions)));
return filter.toString();
}
+ /**
+ * Renders the file extension predicate. A comma separated value matches any one of the
+ * extensions, so a prefix holding more than one file type can be selected in a single sync;
+ * a single value renders exactly the predicate it always did. Empty when no usable extension
+ * is configured, which leaves the filter unchanged.
+ */
+ private static String extensionClause(String objectKey, String extensions) {
+ List<String> predicates = Arrays.stream(extensions.split(","))
+ .map(String::trim)
+ .filter(extension -> !extension.isEmpty())
+ .map(extension -> String.format("%s like '%%%s'", objectKey, extension))
+ .collect(Collectors.toList());
+ if (predicates.isEmpty()) {
Review Comment:
🤖 When the config is set but every entry trims to empty (e.g. `" , "` or `,`), `extensionClause` returns "" so no extension predicate is emitted at all — and because `getPropVal` already returned present, the fallback to the data file format is skipped too. The net effect is that objects of every extension get selected. Is that the intended behavior for a fully-blank list, or would falling back to the default format be safer here so a misconfiguration doesn't silently ingest non-data files?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5364896314
The Azure failure here looks unrelated to this change. The failing phase is `UT spark-datasource DDL & Others`, and the recorded issue is an infrastructure one:
```
Free memory is lower than 5%; Currently used: 95.76%
```
This PR only touches `CloudSourceConfig`, `CloudObjectsSelectorCommon` and its test, all in `hudi-utilities`, so it cannot reach the Scala spark-datasource DDL suite. Needs a re-run rather than a fix.
Reviewed the change itself as part of a pass over the stack and found nothing to raise: `extensionPredicate` renders a byte-identical filter to the previous code for the single-extension case, so existing sources see no behaviour change, and the multi-extension and trimming cases are covered by tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5349382793
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82415395df915019895bb38200cd0753a50aee30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392",
"triggerID" : "82415395df915019895bb38200cd0753a50aee30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82415395df915019895bb38200cd0753a50aee30 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5376600137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82415395df915019895bb38200cd0753a50aee30",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392",
"triggerID" : "82415395df915019895bb38200cd0753a50aee30",
"triggerType" : "PUSH"
}, {
"hash" : "4f57edf018c37497f88731e2bd906c5ee8fa1f72",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f57edf018c37497f88731e2bd906c5ee8fa1f72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82415395df915019895bb38200cd0753a50aee30 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392)
* 4f57edf018c37497f88731e2bd906c5ee8fa1f72 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5376911596
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82415395df915019895bb38200cd0753a50aee30",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392",
"triggerID" : "82415395df915019895bb38200cd0753a50aee30",
"triggerType" : "PUSH"
}, {
"hash" : "4f57edf018c37497f88731e2bd906c5ee8fa1f72",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16475",
"triggerID" : "4f57edf018c37497f88731e2bd906c5ee8fa1f72",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f57edf018c37497f88731e2bd906c5ee8fa1f72 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16475)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19672:
URL: https://github.com/apache/hudi/pull/19672#discussion_r3834342554
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/CloudSourceConfig.java:
##########
@@ -131,7 +131,9 @@ public class CloudSourceConfig extends HoodieConfig {
.noDefaultValue()
.withAlternatives(DELTA_STREAMER_CONFIG_PREFIX + "source.cloud.data.select.file.extension")
.markAdvanced()
- .withDocumentation("Only match files with this extension. By default, this is the same as hoodie.streamer.source.hoodieincr.file.format");
+ .withDocumentation("Only match files with this extension. Accepts a comma separated list, "
Review Comment:
is this true did you check to see this matches behavior of "hoodie.streamer.source.hoodieincr.file.format"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-5348974440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "82415395df915019895bb38200cd0753a50aee30",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392",
"triggerID" : "82415395df915019895bb38200cd0753a50aee30",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 82415395df915019895bb38200cd0753a50aee30 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16392)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(utilities): select more than one file extension in cloud incremental sources [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #19672:
URL: https://github.com/apache/hudi/pull/19672#issuecomment-6701308379
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org