You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xuzifu666 (via GitHub)" <gi...@apache.org> on 2023/10/13 14:32:09 UTC

[I] [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc [hudi]

xuzifu666 opened a new issue, #9858:
URL: https://github.com/apache/hudi/issues/9858

   **_Tips before filing an issue_**
   
   - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
   
   - Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
   
   - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
   
   **Describe the problem you faced**
   
   [A clear and concise description of the problem.](https://issues.apache.org/jira/browse/HUDI-6941)
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1.
   2.
   3.
   4.
   
   **Expected behavior**
   
   A clear and concise description of what you expected to happen.
   
   **Environment Description**
   
   * Hudi version :
   
   * Spark version :
   
   * Hive version :
   
   * Hadoop version :
   
   * Storage (HDFS/S3/GCS..) :
   
   * Running on Docker? (yes/no) :
   
   
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ```Add the stacktrace of the error.```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc [hudi]

Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on issue #9858:
URL: https://github.com/apache/hudi/issues/9858#issuecomment-1762587784

   > @xuzifu666 @wecharyu Thanks for highlighting the issue. This is indeed a regression. The code block
   > 
   > https://github.com/apache/hudi/blob/e6be39661a0aaed4227f8bcbec54e0abaf97fb22/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala#L471-L474
   > 
   > is the main cause. It should be further special cased and not simply depend on number of partition columns. I am going to revert the eager listing first and then put in a proper fix, which should also take care of [HUDI-6320](https://issues.apache.org/jira/browse/HUDI-6320). Additionally, we need some tests to assert spark DAGs for reads much like `TestSparkDataSourceDAGExecution`.
   
   ok，thx and I submit a quick fix https://github.com/apache/hudi/pull/9862，but can not confirm whether cause other risk@codope 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc [hudi]

Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on issue #9858:
URL: https://github.com/apache/hudi/issues/9858#issuecomment-1761784091

   looking into this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #9858:
URL: https://github.com/apache/hudi/issues/9858#issuecomment-1762518643

   @xuzifu666 Thanks for highlighting the issue. This is indeed a regression. The code block https://github.com/apache/hudi/blob/e6be39661a0aaed4227f8bcbec54e0abaf97fb22/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala#L471-L474 is the main cause. It should be further special cased and not simply depend on number of partition columns. I am going to revert the eager listing first and then put in a proper fix, which should also take care of HUDI-6320. Additionally, we need some tests to assert spark DAGs for reads much like `TestSparkDataSourceDAGExecution`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #9858: [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc
URL: https://github.com/apache/hudi/issues/9858


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc [hudi]

Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on issue #9858:
URL: https://github.com/apache/hudi/issues/9858#issuecomment-1761813411

   Eager listing all partition files in file index cause it. It seems not a good idea to get all files in initialization especially when the files are huge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc [hudi]

Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #9858:
URL: https://github.com/apache/hudi/issues/9858#issuecomment-1762624328

   Thanks @xuzifu666. I have fixed some tests and other issues in #9863 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org