You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/06/27 12:03:19 UTC

[GitHub] [hudi] bkosuru opened a new issue, #5983: [SUPPORT] No partition pruning for Incremental Query

bkosuru opened a new issue, #5983:
URL: https://github.com/apache/hudi/issues/5983

   Partition pruning does not seem to be working for Incremental Query. 
   Data Size 1TB 
   Table type COW
   Data in HDFS
   
   Regular query : 8.4 mins
   Incremental Query: 90 mins for the same data
   
   Noticed this in 0.10.1. Not sure if this happens in older versions
   I set the following option on the reader.
   .option("hoodie.file.index.enable", false)
   
   The writer config we use is here - https://github.com/apache/hudi/issues/5741


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan closed issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #5983: [SUPPORT] No partition pruning for Incremental Query
URL: https://github.com/apache/hudi/issues/5983


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] yihua commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1169362980

   @bkosuru Thanks for reporting the issue.  I'll look into that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bkosuru commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
bkosuru commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1169858579

   Thanks for looking into this. I can reproduce the issue in 0.11.1 as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1216263504

   thanks for reaching out! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1216261077

   @yihua : this is already completed right ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1216263332

   we have a tracking [jira](https://issues.apache.org/jira/browse/HUDI-4345) and a [patch](https://github.com/apache/hudi/pull/6045) put up around this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bkosuru commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
bkosuru commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1167320212

   Same issue in 0.8.0 version


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] yihua commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1174651444

   @bkosuru I'm able to reproduce this issue.  I'm working on a fix to enable partition pruning and projection for the incremental query on COW table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bkosuru commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by GitBox <gi...@apache.org>.
bkosuru commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1167274328

   Here is screenshots:
   ![Screen Shot 2022-06-27 at 8 10 53 AM](https://user-images.githubusercontent.com/7408351/175938204-62228f24-6a7f-4a61-abcb-af347643482f.png)
   ![Screen Shot 2022-06-27 at 8 10 44 AM](https://user-images.githubusercontent.com/7408351/175938230-b6724a04-aa24-483a-a58b-44263e9dd311.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] psendyk commented on issue #5983: [SUPPORT] No partition pruning for Incremental Query

Posted by "psendyk (via GitHub)" <gi...@apache.org>.
psendyk commented on issue #5983:
URL: https://github.com/apache/hudi/issues/5983#issuecomment-1720231317

   @yihua @nsivabalan are there plans to support partition pruning for incremental queries on MOR table? We're running into the same performance issue as our queries heavily rely on partition pruning.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org