You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "srinivasst (via GitHub)" <gi...@apache.org> on 2024/10/15 09:51:41 UTC
[I] Qualification Tool for Gluten [incubator-gluten]
srinivasst opened a new issue, #7544:
URL: https://github.com/apache/incubator-gluten/issues/7544
### Description
The purpose of this enhancement is to create a qualification tool that analyzes customer event files to determine which workloads are suitable for execution with Gluten. This is crucial when onboarding new customers, as not all workloads benefit from Gluten's native acceleration—especially workloads with RDD operations, unsupported SQL operators or UDF workloads.
# Proposed Solution:
Develop a Java program to analyze the event files, given a Hadoop file path as input. The program will generate two reports:
- Application Report:
- Percentage of RDD usage
- Percentage of Unsupported SQL operations
- Percentage of supported SQL operations
- Cumulative task time for each application
- Recommendation to use Gluten Acceleration (Recommended if Percentage of supported SQL operations >= 70%)
- Unsupported Operator Report:
- Unsupported SQL operators
- Impact on cumulative CPU time
## Requirements:
- Compatibility with Hadoop file paths that point to:
- Single event files
- Event directories with rolling event files
- Deeply nested directories containing event files
- Compressed event files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [Core] Qualification Tool for Gluten [incubator-gluten]
Posted by "LoseYSelf (via GitHub)" <gi...@apache.org>.
LoseYSelf commented on issue #7544:
URL: https://github.com/apache/incubator-gluten/issues/7544#issuecomment-2422016248
([spark-rapids-tools](https://github.com/NVIDIA/spark-rapids-tools)) is NVIDIA RAPIDS tool, has similar functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Qualification Tool for Gluten [incubator-gluten]
Posted by "xumingming (via GitHub)" <gi...@apache.org>.
xumingming commented on issue #7544:
URL: https://github.com/apache/incubator-gluten/issues/7544#issuecomment-2413527545
Interesting idea, Looking forward to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [Core] Qualification Tool for Gluten [incubator-gluten]
Posted by "surnaik (via GitHub)" <gi...@apache.org>.
surnaik commented on issue #7544:
URL: https://github.com/apache/incubator-gluten/issues/7544#issuecomment-2484782011
You may give it a try here while @srinivasst works on open sourcing this - https://cloud.google.com/dataproc-serverless/docs/guides/native-query-execution#native_query_execution_qualification_tool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [Core] Qualification Tool for Gluten [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW closed issue #7544: [Core] Qualification Tool for Gluten
URL: https://github.com/apache/incubator-gluten/issues/7544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Qualification Tool for Gluten [incubator-gluten]
Posted by "srinivasst (via GitHub)" <gi...@apache.org>.
srinivasst commented on issue #7544:
URL: https://github.com/apache/incubator-gluten/issues/7544#issuecomment-2413427360
WIP - will upload approach and PR in a few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org