You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "prathit06 (via GitHub)" <gi...@apache.org> on 2024/04/29 04:29:54 UTC
[I] [Improvement] Difference in performance of STS & Kyuubi thrift server [kyuubi]
prathit06 opened a new issue, #6345:
URL: https://github.com/apache/kyuubi/issues/6345
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
While trying to use kyuubi with tableau through thrift server exposed by kyuubi, we have noticed that for small dataset everything works fine. But for larger dataset when the collect operation is called, data transfer fails due to driver going OOM. For the same reason kyuubi exposes `kyuubi.operation.incremental.collect` flag which can be set to `true` for incremental collect. While this looks like an ideal solution (which it is) but there are performance bottlenecks when using this flag.
I ran same query through STS & Kyuubi thrift server
Query data size :
- Data size : ~5Gbs
- Records count : 32854241
Results as below with `spark.sql.shuffle.partitions` = 50 :
- STS took 12-15 mins to run the job & transfer the data to tableau.
- Kyuubi thrift was left to run for 2+ hours & transfered data size was only ~600Mbs
As can be seen there is significant performance difference between the two.
Kyuubi version : 1.9
Spark Version : 3.1.2
Running kyuubi on AWS EMR (version 6.5.0) only on primary node
`kyuubi-defaults.conf` config
kyuubi.ha.addresses <hostname>.<aws-region>.compute.internal
kyuubi.operation.incremental.collect true
spark.submit.deployMode cluster # ( have tried with client as well )
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.scheduler.mode FAIR
spark.rdd.compress true
spark.shuffle.service.enabled true
spark.sql.hive.convertMetastoreParquet false
spark.sql.catalogImplementation hive
spark.sql.shuffle.partitions 50
spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
spark.jars <hudi jars> # not required to test as can be done with parquet format as well
spark.kryoserializer.buffer.max 1g
spark.driver.maxResultSize 25g
spark.driver.memory 35g
spark.executor.memory 25g
spark.driver.memoryOverhead 4g
spark.executor.memoryOverhead 3g
spark.cleaner.periodicGC.interval 10min
### How should we improve?
Upon looks at STS & Kyuubi code, i could see a lot of similarities but also differences here & there. One major point i noticed is that some logs were missing from kyuubi, which could help to notice what is happening in kyuubi during data transfer.
For e.g.
Below logs were getting printed in STS but not in kyuubi
```24/04/24 06:49:51 INFO SparkExecuteStatementOperation: Received getNextRowSet request order=FETCH_NEXT and maxRowsL=10000 with a1d07d3a-d6bb-4706-99c0-728fa8115816
24/04/24 06:49:52 INFO SparkExecuteStatementOperation: Returning result set with 10000 rows from offsets [1320000, 1330000) with a1d07d3a-d6bb-4706-99c0-728fa8115816```
To investigate further, maybe adding more logs is a good start to check what exactly is happening & where is the bottleneck.
Please feel free to suggest or ask for any other additional information if needed.
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Improvement] Difference in performance of STS & Kyuubi thrift server [kyuubi]
Posted by "ryze124 (via GitHub)" <gi...@apache.org>.
ryze124 commented on issue #6345:
URL: https://github.com/apache/kyuubi/issues/6345#issuecomment-2975756561
do not use kyuubi.operation.incremental.collect!
if enable this feature, kyuubi will fetch result from executor.
Saving big result to hdfs is better than incremental collect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Improvement] Difference in performance of STS & Kyuubi thrift server [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6345:
URL: https://github.com/apache/kyuubi/issues/6345#issuecomment-2081871497
Hello @prathit06,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Improvement] Difference in performance of STS & Kyuubi thrift server [kyuubi]
Posted by "lanklaas (via GitHub)" <gi...@apache.org>.
lanklaas commented on issue #6345:
URL: https://github.com/apache/kyuubi/issues/6345#issuecomment-2597466273
I am also running into this now. While testing I noticed that on the client side it waits about 60 seconds before kyuubi sends the next batch. I tested using the spark odbc driver as I think this is what tableu uses.
The 60 seconds might just be due to my hardware, but it does look like some kind of polling loop waiting a certain amount of time.
Testing on spark with thrift with incremental collect enabled the batches are sent almost immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Improvement] Difference in performance of STS & Kyuubi thrift server [kyuubi]
Posted by "lanklaas (via GitHub)" <gi...@apache.org>.
lanklaas commented on issue #6345:
URL: https://github.com/apache/kyuubi/issues/6345#issuecomment-2603783279
I tested this with kyuubi 1.10.1 and spark (3.5.4 and 3.4.4) and it seems like the issue is resolved with the latest kyuubi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org