You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dragisa Krsmanovic <dr...@ticketfly.com> on 2016/07/06 17:05:23 UTC
Difference between DataFrame.write.jdbc and DataFrame.write.format("jdbc")
I was expecting to get the same results with both:
dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
and
dataFrame.write.mode(SaveMode.Overwrite).format("jdbc").options(opts).option("dbtable",
"my_table")
In the first example, it behaves as expected. It creates a new table and
populates it with the rows from DataFrame.
In the second case, I get exception:
org.apache.spark.sql.execution.datasources.jdbc.DefaultSource does not
allow create table as select.
Looking at the Spark source, it looks like there is a completely separate
implementation for format("jdbc") and for jdbc(...).
I find that confusing. Unfortunately documentation is rather sparse and one
finds this discrepancy only through trial and error.
Is there a plan to deprecate one of the forms ? Or to allow same
functionality for both ?
I tried both 1.6 and 2.0-preview
--
Dragiša Krsmanović | Platform Engineer | Ticketfly
dragisak@ticketfly.com
@ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog |
facebook.com/ticketfly
Re: Difference between DataFrame.write.jdbc and DataFrame.write.format("jdbc")
Posted by Xiao Li <ga...@gmail.com>.
Hi, Dragisa,
Just submitted a PR for implementing the save API.
https://github.com/apache/spark/pull/14077
Let me know if you have any question,
Xiao
2016-07-06 10:41 GMT-07:00 Rabin Banerjee <de...@gmail.com>:
> HI Buddy,
>
> I sued both but DataFrame.write.jdbc is old, and will work if provide
> table name , It wont work if you provide custom queries . Where
> as DataFrame.write.format is more generic as well as working perfectly with
> not only table name but also custom queries . Hence I recommend to use
> the DataFrame.write.format("jdbc") .
>
> Cheers !
> Rabin
>
>
>
> On Wed, Jul 6, 2016 at 10:35 PM, Dragisa Krsmanovic <
> dragisak@ticketfly.com> wrote:
>
>> I was expecting to get the same results with both:
>>
>> dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
>>
>> and
>>
>> dataFrame.write.mode(SaveMode.Overwrite).format("jdbc").options(opts).option("dbtable",
>> "my_table")
>>
>>
>> In the first example, it behaves as expected. It creates a new table and
>> populates it with the rows from DataFrame.
>>
>> In the second case, I get exception:
>> org.apache.spark.sql.execution.datasources.jdbc.DefaultSource does not
>> allow create table as select.
>>
>> Looking at the Spark source, it looks like there is a completely separate
>> implementation for format("jdbc") and for jdbc(...).
>>
>> I find that confusing. Unfortunately documentation is rather sparse and
>> one finds this discrepancy only through trial and error.
>>
>> Is there a plan to deprecate one of the forms ? Or to allow same
>> functionality for both ?
>>
>> I tried both 1.6 and 2.0-preview
>> --
>>
>> Dragiša Krsmanović | Platform Engineer | Ticketfly
>>
>> dragisak@ticketfly.com
>>
>> @ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog |
>> facebook.com/ticketfly
>>
>
>
Re: Difference between DataFrame.write.jdbc and DataFrame.write.format("jdbc")
Posted by Rabin Banerjee <de...@gmail.com>.
HI Buddy,
I sued both but DataFrame.write.jdbc is old, and will work if provide
table name , It wont work if you provide custom queries . Where
as DataFrame.write.format is more generic as well as working perfectly with
not only table name but also custom queries . Hence I recommend to use
the DataFrame.write.format("jdbc") .
Cheers !
Rabin
On Wed, Jul 6, 2016 at 10:35 PM, Dragisa Krsmanovic <dr...@ticketfly.com>
wrote:
> I was expecting to get the same results with both:
>
> dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
>
> and
>
> dataFrame.write.mode(SaveMode.Overwrite).format("jdbc").options(opts).option("dbtable",
> "my_table")
>
>
> In the first example, it behaves as expected. It creates a new table and
> populates it with the rows from DataFrame.
>
> In the second case, I get exception:
> org.apache.spark.sql.execution.datasources.jdbc.DefaultSource does not
> allow create table as select.
>
> Looking at the Spark source, it looks like there is a completely separate
> implementation for format("jdbc") and for jdbc(...).
>
> I find that confusing. Unfortunately documentation is rather sparse and
> one finds this discrepancy only through trial and error.
>
> Is there a plan to deprecate one of the forms ? Or to allow same
> functionality for both ?
>
> I tried both 1.6 and 2.0-preview
> --
>
> Dragiša Krsmanović | Platform Engineer | Ticketfly
>
> dragisak@ticketfly.com
>
> @ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog |
> facebook.com/ticketfly
>
Re: Difference between DataFrame.write.jdbc and DataFrame.write.format("jdbc")
Posted by Dragisa Krsmanovic <dr...@ticketfly.com>.
Yes, I had the save() at the end. I truncated example to highlight the
difference and forgot to put back the save()
It would be great to have the same behavior (and same code used) for both
jdbc() and format("jdbc").
Thank you.
On Wed, Jul 6, 2016 at 10:21 AM, Xiao Li <ga...@gmail.com> wrote:
> Hi, Dragisa,
>
> Your second way is incomplete, right? To get the error you showed, you
> need to put save() there.
>
> Yeah, we can implement the trait CreatableRelationProvider for JDBC. Then,
> you will not see that error.
>
> Will submit a PR for that.
>
> Thanks,
>
> Xiao
>
>
> 2016-07-06 10:05 GMT-07:00 Dragisa Krsmanovic <dr...@ticketfly.com>:
>
>> I was expecting to get the same results with both:
>>
>> dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
>>
>> and
>>
>> dataFrame.write.mode(SaveMode.Overwrite).format("jdbc").options(opts).option("dbtable",
>> "my_table")
>>
>>
>> In the first example, it behaves as expected. It creates a new table and
>> populates it with the rows from DataFrame.
>>
>> In the second case, I get exception:
>> org.apache.spark.sql.execution.datasources.jdbc.DefaultSource does not
>> allow create table as select.
>>
>> Looking at the Spark source, it looks like there is a completely separate
>> implementation for format("jdbc") and for jdbc(...).
>>
>> I find that confusing. Unfortunately documentation is rather sparse and
>> one finds this discrepancy only through trial and error.
>>
>> Is there a plan to deprecate one of the forms ? Or to allow same
>> functionality for both ?
>>
>> I tried both 1.6 and 2.0-preview
>> --
>>
>> Dragiša Krsmanović | Platform Engineer | Ticketfly
>>
>> dragisak@ticketfly.com
>>
>> @ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog |
>> facebook.com/ticketfly
>>
>
>
--
Dragiša Krsmanović | Platform Engineer | Ticketfly
dragisak@ticketfly.com
@ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog |
facebook.com/ticketfly
Re: Difference between DataFrame.write.jdbc and DataFrame.write.format("jdbc")
Posted by Xiao Li <ga...@gmail.com>.
Hi, Dragisa,
Your second way is incomplete, right? To get the error you showed, you need
to put save() there.
Yeah, we can implement the trait CreatableRelationProvider for JDBC. Then,
you will not see that error.
Will submit a PR for that.
Thanks,
Xiao
2016-07-06 10:05 GMT-07:00 Dragisa Krsmanovic <dr...@ticketfly.com>:
> I was expecting to get the same results with both:
>
> dataFrame.write.mode(SaveMode.Overwrite).jdbc(dbUrl, "my_table", props)
>
> and
>
> dataFrame.write.mode(SaveMode.Overwrite).format("jdbc").options(opts).option("dbtable",
> "my_table")
>
>
> In the first example, it behaves as expected. It creates a new table and
> populates it with the rows from DataFrame.
>
> In the second case, I get exception:
> org.apache.spark.sql.execution.datasources.jdbc.DefaultSource does not
> allow create table as select.
>
> Looking at the Spark source, it looks like there is a completely separate
> implementation for format("jdbc") and for jdbc(...).
>
> I find that confusing. Unfortunately documentation is rather sparse and
> one finds this discrepancy only through trial and error.
>
> Is there a plan to deprecate one of the forms ? Or to allow same
> functionality for both ?
>
> I tried both 1.6 and 2.0-preview
> --
>
> Dragiša Krsmanović | Platform Engineer | Ticketfly
>
> dragisak@ticketfly.com
>
> @ticketfly <https://twitter.com/ticketfly> | ticketfly.com/blog |
> facebook.com/ticketfly
>