You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Noah-FetchRewards (via GitHub)" <gi...@apache.org> on 2025/02/16 23:03:50 UTC
[I] Need help running benchmarks and other pyspark jobs. [datafusion-comet]
Noah-FetchRewards opened a new issue, #1411:
URL: https://github.com/apache/datafusion-comet/issues/1411
### Describe the bug
I am trying to get a pyspark job to run on apache comet using an EKS cluster; however after 20+ hours, I am unable to do so for a variety of reasons.
Explanation:
I tried to follow the example used in the benchmark https://github.com/apache/datafusion-comet/tree/main/benchmarks, where I built the image locally, got it uploaded to ECR, then tried to use the spark submit command to run the tpcbench.py (after updating the .jar files to point to the latest version); however, the sample data to run the data at,
conf spark.kubernetes.executor.volumes.hostPath.tpcdata.options.path=/mnt/bigdata/tpcds/sf100/
is obviously not in the image and I gave up pretty quickly, as I am unsure how to get the data loaded into the container itself.
I then tried to run the pyspark files that were already locally installed into the base datafusion-comet image, specifically:
local:///opt/spark/examples/src/main/python/pi.py
However, I would just see the message:
25/02/16 17:42:12 INFO LoggingPodStatusWatcherImpl: Application status for spark-4a588861176e45218dc96d608820b902 (phase: Running)
Run indefinitely, and it never seems to actually complete, so I can't seem to confirm if pyspark jobs actually work on Apache comet.
Using the command:
`
spark-submit \
--master $SPARK_MASTER \
--deploy-mode cluster \
--name comet-tpcbench \
--driver-memory 8G \
--conf spark.driver.memory=8G \
--conf spark.executor.instances=1 \
--conf spark.executor.memory=32G \
--conf spark.executor.cores=8 \
--conf spark.cores.max=8 \
--conf spark.task.cpus=1 \
--conf spark.executor.memoryOverhead=3G \
--jars local://$COMET_JAR \
--conf spark.executor.extraClassPath=$COMET_JAR \
--conf spark.driver.extraClassPath=$COMET_JAR \
--conf spark.plugins=org.apache.spark.CometPlugin \
--conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
--conf spark.comet.enabled=true \
--conf spark.comet.exec.enabled=true \
--conf spark.comet.exec.all.enabled=true \
--conf spark.comet.cast.allowIncompatible=true \
--conf spark.comet.exec.shuffle.enabled=true \
--conf spark.comet.exec.shuffle.mode=auto \
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
--conf spark.kubernetes.namespace=default \
--conf spark.kubernetes.driver.pod.name=tpcbench \
--conf spark.kubernetes.container.image=$COMET_DOCKER_IMAGE \
local:///opt/spark/examples/src/main/python/pi.py
`
I also tried to run the examples on https://datafusion.apache.org/comet/user-guide/kubernetes.html using the spark-operator (after fixing the outdated jars again), but the job seems to run for a while and not complete or give me logs.
I'm not particularly experienced with spark on kuberentes, but how exactly am I supposed to run a job to completion using apache comet at all?
I even bothered to install microk8s, and tried to follow the examples to run it there, to no avail.
I would also appreciate links to useful reference materials to getting started doing this sort of work.
I'd like to not I also spent 2 days trying to get apache ballista to work with the existing examples, only to also run into a plethora of strange bugs as well, I was hoping Comet would be alot easier to be productive on.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Need help running benchmarks and other pyspark jobs. [datafusion-comet]
Posted by "Noah-FetchRewards (via GitHub)" <gi...@apache.org>.
Noah-FetchRewards commented on issue #1411:
URL: https://github.com/apache/datafusion-comet/issues/1411#issuecomment-2664570522
Thanks @andygrove, your suggestion of disable comet features and running on Spark first, then reenabling worked for me.
For some reason disabling the documentation suggest flags
"spark.comet.cast.allowIncompatible": "false"
"spark.comet.exec.shuffle.enabled": "false"
caused my Comet jobs to actually run to completion, when previously it ran indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Need help running benchmarks and other pyspark jobs. [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1411:
URL: https://github.com/apache/datafusion-comet/issues/1411#issuecomment-2663580349
Ho @Noah-FetchRewards. It looks like you are trying to do two things at once - run TPC-H on Spark on k8s, and run Comet.
If your requirement is to run in k8s then I would recommend getting regular Spark running in k8s first, and then enable Comet.
Spark has very good documentation on running with k8s:
https://spark.apache.org/docs/3.5.4/running-on-kubernetes.html
> however, the sample data to run the data at,
conf spark.kubernetes.executor.volumes.hostPath.tpcdata.options.path=/mnt/bigdata/tpcds/sf100/
is obviously not in the image and I gave up pretty quickly, as I am unsure how to get the data loaded into the container itself.
I would not recommend loading data into the container. You would typically want to use a PersistentVolumeClaim if you want to store data directly on host machines, or more typically, you could read the data from an S3 bucket or similar. This isn't specific to Comet.
If you want to run locally on a single node i.e. laptop/desktop then I would recommend running Spark in standalone mode.
I hope these are helpful suggestions. I'm happy to help provide more guidance if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Need help running benchmarks and other pyspark jobs. [datafusion-comet]
Posted by "Noah-FetchRewards (via GitHub)" <gi...@apache.org>.
Noah-FetchRewards closed issue #1411: Need help running benchmarks and other pyspark jobs.
URL: https://github.com/apache/datafusion-comet/issues/1411
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org