You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ravi Hemnani <ra...@gmail.com> on 2014/02/11 17:31:54 UTC
Query execution in spark
Hey,
Can anyone explain how the job basically runs in spark?
The number of mapper, reducers, the tmp files created and which tmp file
contains what data and how to set the number of reducer tasks as we do in
hadoop.
This would prove to be a big help. Thank you.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Query execution in spark
Posted by Ognen Duzlevski <og...@plainvanillagames.com>.
On 2/11/14, 12:57 PM, Mayur Rustagi wrote:
> I have found this quite useful
> http://www.youtube.com/watch?v=49Hr5xZyTEA
Thank you!
Ognen
Re: Query execution in spark
Posted by Ravi Hemnani <ra...@gmail.com>.
On 02/12/2014 12:29 AM, Mayur Rustagi [via Apache Spark User List] wrote:
> I have found this quite useful
> http://www.youtube.com/watch?v=49Hr5xZyTEA
>
>
> Mayur Rustagi
> Ph: +919632149971
> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
> <http://www.sigmoidanalytics.com>
> https://twitter.com/mayur_rustagi
>
>
>
> On Tue, Feb 11, 2014 at 10:16 AM, Andrew Ash <[hidden email]
> </user/SendEmail.jtp?type=node&node=1405&i=0>> wrote:
>
> Hi Ravi,
>
> Have you read through the docs? I'm not sure there's a page that
> directly answers your question but this one gives you an overview
> of the cluster.
>
> http://spark.incubator.apache.org/docs/latest/cluster-overview.html
>
> Andrew
>
>
> On Tue, Feb 11, 2014 at 8:31 AM, Ravi Hemnani <[hidden email]
> </user/SendEmail.jtp?type=node&node=1405&i=1>> wrote:
>
> Hey,
>
> Can anyone explain how the job basically runs in spark?
>
> The number of mapper, reducers, the tmp files created and
> which tmp file
> contains what data and how to set the number of reducer tasks
> as we do in
> hadoop.
>
> This would prove to be a big help. Thank you.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390.html
> Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
>
>
>
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390p1405.html
>
> To unsubscribe from Query execution in spark, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1390&code=cmF2aWlpaGVtbmFuaUBnbWFpbC5jb218MTM5MHwtMTk4NTIxNTA1Nw==>.
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
I was watching the same video and it made me more curious about various
things.
Currently i am working on what things i need to look into for
optimization of my spark cluster. Ill share my notes with you soon. :)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390p1423.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Query execution in spark
Posted by Mayur Rustagi <ma...@gmail.com>.
I have found this quite useful
http://www.youtube.com/watch?v=49Hr5xZyTEA
Mayur Rustagi
Ph: +919632149971
h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
https://twitter.com/mayur_rustagi
On Tue, Feb 11, 2014 at 10:16 AM, Andrew Ash <an...@andrewash.com> wrote:
> Hi Ravi,
>
> Have you read through the docs? I'm not sure there's a page that directly
> answers your question but this one gives you an overview of the cluster.
>
> http://spark.incubator.apache.org/docs/latest/cluster-overview.html
>
> Andrew
>
>
> On Tue, Feb 11, 2014 at 8:31 AM, Ravi Hemnani <ra...@gmail.com>wrote:
>
>> Hey,
>>
>> Can anyone explain how the job basically runs in spark?
>>
>> The number of mapper, reducers, the tmp files created and which tmp file
>> contains what data and how to set the number of reducer tasks as we do in
>> hadoop.
>>
>> This would prove to be a big help. Thank you.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
Re: Query execution in spark
Posted by Andrew Ash <an...@andrewash.com>.
Hi Ravi,
Have you read through the docs? I'm not sure there's a page that directly
answers your question but this one gives you an overview of the cluster.
http://spark.incubator.apache.org/docs/latest/cluster-overview.html
Andrew
On Tue, Feb 11, 2014 at 8:31 AM, Ravi Hemnani <ra...@gmail.com>wrote:
> Hey,
>
> Can anyone explain how the job basically runs in spark?
>
> The number of mapper, reducers, the tmp files created and which tmp file
> contains what data and how to set the number of reducer tasks as we do in
> hadoop.
>
> This would prove to be a big help. Thank you.
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Query-execution-in-spark-tp1390.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>