[I] Cloud incremental sources cannot select more than one file extension [hudi]

["rahil-c (via GitHub)" <gi...@apache.org>]

rahil-c opened a new issue, #19671:
URL: https://github.com/apache/hudi/issues/19671

   ### Problem
   
   `S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` can only select one file
   extension per pipeline. `hoodie.streamer.source.cloud.data.select.file.extension` is
   interpolated into a single SQL predicate, so a bucket holding more than one file type
   cannot be ingested by one streamer.
   
   Setting a list looks like it should work, and fails quietly. With
   `select.file.extension=json,jsonl` the generated filter becomes:
   
   ```
   s3.object.size > 0 and s3.object.key like '%json,jsonl'
   ```
   
   That is a literal match on the comma joined string, so it matches nothing. The job
   succeeds, commits nothing, and reports no error, which is hard to diagnose because
   there is no signal that the filter is at fault.
   
   ### Expected
   
   A comma separated value should select files matching any one of the extensions, while a
   single value keeps behaving exactly as it does today.
   
   ### Notes
   
   The same config also falls back to the data file format when unset, which defaults to
   `parquet`. That behaviour is long standing and remains unchanged here, but it is
   currently undocumented and worth pinning with a test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Cloud incremental sources cannot select more than one file extension [hudi]

["rahil-c (via GitHub)" <gi...@apache.org>]

rahil-c commented on issue #19671:
URL: https://github.com/apache/hudi/issues/19671#issuecomment-5348846112

   PR: #19672


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org