You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "lgbo-ustc (via GitHub)" <gi...@apache.org> on 2024/11/19 07:21:34 UTC

[I] [CH] There is performance regression with lazy expand enable when there are high cardinality grouping keys [incubator-gluten]

lgbo-ustc opened a new issue, #7986:
URL: https://github.com/apache/incubator-gluten/issues/7986

   ### Backend
   
   CH (ClickHouse)
   
   ### Bug description
   
   
   Lazy expand(#7647) could work will on low cardinality group keys, but there is performance regression when there are high cardinality grouping keys. For example, tpcds q67 runs slower, since the `i_product_name` is a high cardinality column. It's a problem we have discuss in #7647, let's see what we could do to improve this cases.
   
   - disable lazy expand 
   
   ![image](https://github.com/user-attachments/assets/e7be0fa6-7c97-4d9d-99e0-d799cee902f3)
   
   - enable lazy expand
   ![image](https://github.com/user-attachments/assets/79e128ed-cb89-44dd-be70-289f7e68edd6)
   
   
   Obviously, with lazy expand enable, this first aggregate stage generates more rows. This increase the execution time of shuffle and the second aggregate stage. We use a simple algorithm to decide whether aggregate data from the expand operator, this cause there is no rows are aggregated from the expand operator. 
   
   
   
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] There is performance regression with lazy expand enable when there are high cardinality grouping keys [incubator-gluten]

Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #7986:
URL: https://github.com/apache/incubator-gluten/issues/7986#issuecomment-2485513652

   About the performance degradation in `sink`, I guess it's related to the blocks number growth. The total input rows are equal, but they are stored into more blocks with lazy expand enable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] There is performance regression with lazy expand enable when there are high cardinality grouping keys [incubator-gluten]

Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #7986:
URL: https://github.com/apache/incubator-gluten/issues/7986#issuecomment-2484993782

   
   - disable lazy expand
   ![image](https://github.com/user-attachments/assets/81b7e6f8-217c-4fbf-8a11-496175d42f36)
   - enable lazy expand
   ![image](https://github.com/user-attachments/assets/c0df34ec-196a-447c-9d4b-b2b2ab9d93a1)
   
   The time of sink increases significantly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] There is performance regression with lazy expand enable when there are high cardinality grouping keys [incubator-gluten]

Posted by "zzcclp (via GitHub)" <gi...@apache.org>.
zzcclp closed issue #7986: [CH] There is performance regression with lazy expand enable when there are high cardinality grouping keys
URL: https://github.com/apache/incubator-gluten/issues/7986


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [CH] There is performance regression with lazy expand enable when there are high cardinality grouping keys [incubator-gluten]

Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on issue #7986:
URL: https://github.com/apache/incubator-gluten/issues/7986#issuecomment-2485319462

   q67的情况比较特殊。在使用全部的grouping keys进行聚合时,结果的行数和输入的行数基本一样。这就造成整体的效果和expand先执行的效果一样。因为两种方法需要在hash map中查找插入的数据量是一样的。
   ![image](https://github.com/user-attachments/assets/b76fe578-61fb-4956-94e7-43b2060c3c91)
   
   
   只有全部grouping keys的聚合结果能一定程度比输入的行数少时,才会有性能提升。
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org