You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by sujeet jog <su...@gmail.com> on 2016/07/21 14:59:15 UTC
Load selected rows with sqlContext in the dataframe
I have a table of size 5GB, and want to load selective rows into dataframe
instead of loading the entire table in memory,
For me memory is a constraint hence , and i would like to peridically load
few set of rows and perform dataframe operations on it,
,
for the "dbtable" is there a way to perform select * from master_schema
where 'TID' = '100_0';
which can load only this to memory as dataframe .
Currently I'm using code as below
val df = sqlContext.read .format("jdbc")
.option("url", url)
.option("dbtable", "master_schema").load()
Thansk,
Sujeet
Re: Load selected rows with sqlContext in the dataframe
Posted by sujeet jog <su...@gmail.com>.
Thanks Todd.
On Thu, Jul 21, 2016 at 9:18 PM, Todd Nist <ts...@gmail.com> wrote:
> You can set the dbtable to this:
>
> .option("dbtable", "(select * from master_schema where 'TID' = '100_0')")
>
> HTH,
>
> Todd
>
>
> On Thu, Jul 21, 2016 at 10:59 AM, sujeet jog <su...@gmail.com> wrote:
>
>> I have a table of size 5GB, and want to load selective rows into
>> dataframe instead of loading the entire table in memory,
>>
>>
>> For me memory is a constraint hence , and i would like to peridically
>> load few set of rows and perform dataframe operations on it,
>>
>> ,
>> for the "dbtable" is there a way to perform select * from master_schema
>> where 'TID' = '100_0';
>> which can load only this to memory as dataframe .
>>
>>
>>
>> Currently I'm using code as below
>> val df = sqlContext.read .format("jdbc")
>> .option("url", url)
>> .option("dbtable", "master_schema").load()
>>
>>
>> Thansk,
>> Sujeet
>>
>
>
Re: Load selected rows with sqlContext in the dataframe
Posted by Todd Nist <ts...@gmail.com>.
You can set the dbtable to this:
.option("dbtable", "(select * from master_schema where 'TID' = '100_0')")
HTH,
Todd
On Thu, Jul 21, 2016 at 10:59 AM, sujeet jog <su...@gmail.com> wrote:
> I have a table of size 5GB, and want to load selective rows into dataframe
> instead of loading the entire table in memory,
>
>
> For me memory is a constraint hence , and i would like to peridically load
> few set of rows and perform dataframe operations on it,
>
> ,
> for the "dbtable" is there a way to perform select * from master_schema
> where 'TID' = '100_0';
> which can load only this to memory as dataframe .
>
>
>
> Currently I'm using code as below
> val df = sqlContext.read .format("jdbc")
> .option("url", url)
> .option("dbtable", "master_schema").load()
>
>
> Thansk,
> Sujeet
>