You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2024/03/22 10:18:35 UTC
[PR] [MINOR] Improve SparkPi example [spark]
EnricoMi opened a new pull request, #45664:
URL: https://github.com/apache/spark/pull/45664
### What changes were proposed in this pull request?
- Make `SparkPi` example use Dataset API
- Allow setting partition size
- Print available options
### Why are the changes needed?
1. The `SparkPi` example uses the RDD API, which does not show the "SQL / DataFrame" tab in the Spark UI.
2. The partitions are very small (100k rows), which is not suitable to exemplify compute-expensive workload.
### Does this PR introduce _any_ user-facing change?
Adds one option to the `SparkPi` example.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR] Improve SparkPi example [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45664:
URL: https://github.com/apache/spark/pull/45664#issuecomment-2016995240
According to the structure, it should be RDD/Core example. If we want this, we should put that into `sql`, and fix the link of the example in the docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR] Improve SparkPi example [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #45664:
URL: https://github.com/apache/spark/pull/45664#issuecomment-2016909814
@dongjoon-hyun what do you think about the partitions option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR] Improve SparkPi example [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #45664:
URL: https://github.com/apache/spark/pull/45664#issuecomment-2067958222
Thanks for the input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR] Improve SparkPi example [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #45664:
URL: https://github.com/apache/spark/pull/45664#issuecomment-2067877239
Sorry but I don't think this is a kind of `[MINOR]` category work item and I'm not sure about the contribution of this PR, @EnricoMi . Instead, I believe this example is very important than we think because this is a kind of `HelloWorld`. I'd recommend to keep the example as neat and short as possible in the AS-IS status for our newcomers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR] Improve SparkPi example [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #45664:
URL: https://github.com/apache/spark/pull/45664#issuecomment-2067659018
@dongjoon-hyun https://github.com/apache/spark/pull/45664#issuecomment-2016909814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR] Improve SparkPi example [spark]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi closed pull request #45664: [MINOR] Improve SparkPi example
URL: https://github.com/apache/spark/pull/45664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org