You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 09:18:33 UTC

[I] Only push down all filters to mor base reader [hudi]

hudi-bot opened a new issue, #16301:
URL: https://github.com/apache/hudi/issues/16301

   {code:java}
   val preMergeBaseFileReader = if (isMOR) {
     readerFormat.buildReaderWithPartitionValues(sparkSession, dataSchema, StructType(Seq.empty),
       requiredSchemaWithMandatory, requiredFilters, options, new Configuration(hadoopConf))
   } else {
     _: PartitionedFile => Iterator.empty
   } {code}
   requiredFilters can only have recorkey filters
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-7066
   - Type: Bug
   - Epic: https://issues.apache.org/jira/browse/HUDI-6568


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Only push down all filters to mor base reader [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16301: Only push down all filters to mor base reader
URL: https://github.com/apache/hudi/issues/16301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Only push down all filters to mor base reader [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16301:
URL: https://github.com/apache/hudi/issues/16301#issuecomment-4795573052

   This appears to be addressed in current master. The `preMergeBaseFileReader` code pattern referenced in the issue no longer exists — MOR base reads were refactored into the file group reader path. PR #14183 (commit `ec3b35d6`, "feat: introduce pk filter push-down to base file") added PK-safe filter pushdown via `filterIsSafeForPrimaryKey` in `SparkFileFormatInternalRowReaderContext.scala`. The MOR reader now uses:
   
   ```scala
   private lazy val morFilters = filters.filter(filterIsSafeForPrimaryKey(_, recordKeyFields)) ++ requiredFilters
   ```
   
   so PK-safe optional filters (not only `requiredFilters`) are pushed down to the base reader before merging with logs. Closing as fixed — please reopen if the underlying concern still applies on master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org