Fwd: Spark for Oracle sample code

[Cui Lin <ic...@gmail.com>]

Hello, All,

I found the examples for JDBC connection are mostly read the whole table
and then do operations like joining.

val jdbcDF = sqlContext.read.format("jdbc").options(
  Map("url" -> "jdbc:postgresql:dbserver",
  "dbtable" -> "schema.tablename")).load()


Sometimes it is not practical since the whole table data is too big and not
necessary.

What makes sense to me is to use sparksql to get subset data from oracle
tables using sql-like statement.
I couldn't find such examples. Can someone show me?



-- 
Best regards!

Lin,Cui



-- 
Best regards!

Lin,Cui

Re: Spark for Oracle sample code

[Michael Armbrust <mi...@databricks.com>]

In most cases predicates that you add to jdbcDF will be push down into
oracle, preventing the whole table from being sent over.

df.where("column = 1")

Another common pattern is to save the table to parquet or something for
repeat querying.

Michael

On Fri, Sep 25, 2015 at 3:13 PM, Cui Lin <ic...@gmail.com> wrote:

>
>
> Hello, All,
>
> I found the examples for JDBC connection are mostly read the whole table
> and then do operations like joining.
>
> val jdbcDF = sqlContext.read.format("jdbc").options(
>   Map("url" -> "jdbc:postgresql:dbserver",
>   "dbtable" -> "schema.tablename")).load()
>
>
> Sometimes it is not practical since the whole table data is too big and
> not necessary.
>
> What makes sense to me is to use sparksql to get subset data from oracle
> tables using sql-like statement.
> I couldn't find such examples. Can someone show me?
>
>
>
> --
> Best regards!
>
> Lin,Cui
>
>
>
> --
> Best regards!
>
> Lin,Cui
>