Spark Doubts

[Sid <fl...@gmail.com>]

Hi Team,

I have a few doubts about the below questions:

1) data frame will reside where? memory? disk? memory allocation about data
frame?
2) How do you configure each partition?
3) Is there any way to calculate the exact partitions needed to load a
specific file?

Thanks,
Sid

Re: Spark Doubts

["Apostolos N. Papadopoulos" <pa...@csd.auth.gr>]

Dear Sid.

You are asking questions for which answers exist in the Apache Spark 
website or in books or in MOOCS or in other URLs.

For example, take a look at this one: 
https://sparkbyexamples.com/spark/spark-dataframe-cache-and-persist-explained/
<https://sparkbyexamples.com/spark/spark-dataframe-cache-and-persist-explained/>

https://spark.apache.org/docs/latest/sql-programming-guide.html

What do you mean by question 2?

About question 3, it depends on how you load the file. For example, if 
you have a text file in HDFS and you want to

use an RDD, initially, the number of partitions equals the number of 
HDFS blocks, unless you specify the number of

partitions when you create the RDD from the file.

I would suggest first to go through a book devoted to Spark, like The 
Definitive Guide, or any other similar resource.

Also, I would suggest to take a MOOC on Spark (e.g., in Coursera, edX, 
etc).

All the best,

Apostolos


On 21/6/22 22:16, Sid wrote:
> Hi Team,
>
> I have a few doubts about the below questions:
>
> 1) data frame will reside where? memory? disk? memory allocation about 
> data frame?
> 2) How do you configure each partition?
> 3) Is there any way to calculate the exact partitions needed to load a 
> specific file?
>
> Thanks,
> Sid

-- 
Apostolos N. Papadopoulos, Associate Professor
Department of Informatics
Aristotle University of Thessaloniki
Thessaloniki, GREECE
tel: ++0030312310991918
email: papadopo@csd.auth.gr
twitter: @papadopoulos_ap
web: http://datalab.csd.auth.gr/~apostol


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Spark Doubts

[Sid <fl...@gmail.com>]

Hi,

Thanks for your answers. Much appreciated

I know that we can cache the data frame in memory or disk but I want to
understand when the data frame is loaded initially and where does it reside
by default?


Thanks,
Sid

On Wed, Jun 22, 2022 at 6:10 AM Yong Walt <yo...@gmail.com> wrote:

> These are the basic concepts in spark :)
> You may take a bit time to read this small book:
> https://cloudcache.net/resume/PDDWS2-V2.pdf
>
> regards
>
>
> On Wed, Jun 22, 2022 at 3:17 AM Sid <fl...@gmail.com> wrote:
>
>> Hi Team,
>>
>> I have a few doubts about the below questions:
>>
>> 1) data frame will reside where? memory? disk? memory allocation about
>> data frame?
>> 2) How do you configure each partition?
>> 3) Is there any way to calculate the exact partitions needed to load a
>> specific file?
>>
>> Thanks,
>> Sid
>>
>

Re: Spark Doubts

[Yong Walt <yo...@gmail.com>]

These are the basic concepts in spark :)
You may take a bit time to read this small book:
https://cloudcache.net/resume/PDDWS2-V2.pdf

regards


On Wed, Jun 22, 2022 at 3:17 AM Sid <fl...@gmail.com> wrote:

> Hi Team,
>
> I have a few doubts about the below questions:
>
> 1) data frame will reside where? memory? disk? memory allocation about
> data frame?
> 2) How do you configure each partition?
> 3) Is there any way to calculate the exact partitions needed to load a
> specific file?
>
> Thanks,
> Sid
>