You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yasemin Kaya <go...@gmail.com> on 2016/01/08 16:36:17 UTC
write new data to mysql
Hi,
I want to write dataframe existing mysql table, but when i use
*peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
"track_on_alarm",false)*
it says "Table track_on_alarm already exists."
And when i *use peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
"track_on_alarm",true)*
i lost the existing data.
How i can write new data to db?
Best,
yasemin
--
hiç ender hiç
Re: write new data to mysql
Posted by Todd Nist <ts...@gmail.com>.
Sorry, did not see your update until now.
On Fri, Jan 8, 2016 at 3:52 PM, Todd Nist <ts...@gmail.com> wrote:
> Hi Yasemin,
>
> What version of Spark are you using? Here is the reference, it is off of
> the DataFrame
> https://spark.apache.org/docs/latest/api/java/index.html#org.apache.spark.sql.DataFrame
> and provides a DataFrameWriter,
> https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameWriter.html
> :
>
> DataFrameWriter
> <https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameWriter.html>
> *write
> <https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrame.html#write()>*
> ()
> Interface for saving the content of the DataFrame
> <https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrame.html> out
> into external storage.
>
> It is the very last method defined there in the api docs.
>
> HTH.
>
> -Todd
>
>
> On Fri, Jan 8, 2016 at 2:27 PM, Yasemin Kaya <go...@gmail.com> wrote:
>
>> Hi,
>> There is no write function that Todd mentioned or i cant find it.
>> The code and error are in gist
>> <https://gist.github.com/yaseminn/f5a2b78b126df71dfd0b>. Could you check
>> it out please?
>>
>> Best,
>> yasemin
>>
>> 2016-01-08 18:23 GMT+02:00 Todd Nist <ts...@gmail.com>:
>>
>>> It is not clear from the information provided why the insertIntoJDBC
>>> failed in #2. I would note that method on the DataFrame as been deprecated
>>> since 1.4, not sure what version your on. You should be able to do
>>> something like this:
>>>
>>> DataFrame.write.mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL_WRITE,
>>> "track_on_alarm", connectionProps)
>>>
>>> HTH.
>>>
>>> -Todd
>>>
>>> On Fri, Jan 8, 2016 at 10:53 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Which Spark release are you using ?
>>>>
>>>> For case #2, was there any error / clue in the logs ?
>>>>
>>>> Cheers
>>>>
>>>> On Fri, Jan 8, 2016 at 7:36 AM, Yasemin Kaya <go...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want to write dataframe existing mysql table, but when i use
>>>>> *peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>>>> "track_on_alarm",false)*
>>>>>
>>>>> it says "Table track_on_alarm already exists."
>>>>>
>>>>> And when i *use peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>>>> "track_on_alarm",true)*
>>>>>
>>>>> i lost the existing data.
>>>>>
>>>>> How i can write new data to db?
>>>>>
>>>>> Best,
>>>>> yasemin
>>>>>
>>>>> --
>>>>> hiç ender hiç
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> hiç ender hiç
>>
>
>
Re: write new data to mysql
Posted by Todd Nist <ts...@gmail.com>.
Hi Yasemin,
What version of Spark are you using? Here is the reference, it is off of
the DataFrame
https://spark.apache.org/docs/latest/api/java/index.html#org.apache.spark.sql.DataFrame
and provides a DataFrameWriter,
https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameWriter.html
:
DataFrameWriter
<https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrameWriter.html>
*write
<https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrame.html#write()>*
()
Interface for saving the content of the DataFrame
<https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/DataFrame.html>
out
into external storage.
It is the very last method defined there in the api docs.
HTH.
-Todd
On Fri, Jan 8, 2016 at 2:27 PM, Yasemin Kaya <go...@gmail.com> wrote:
> Hi,
> There is no write function that Todd mentioned or i cant find it.
> The code and error are in gist
> <https://gist.github.com/yaseminn/f5a2b78b126df71dfd0b>. Could you check
> it out please?
>
> Best,
> yasemin
>
> 2016-01-08 18:23 GMT+02:00 Todd Nist <ts...@gmail.com>:
>
>> It is not clear from the information provided why the insertIntoJDBC
>> failed in #2. I would note that method on the DataFrame as been deprecated
>> since 1.4, not sure what version your on. You should be able to do
>> something like this:
>>
>> DataFrame.write.mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL_WRITE,
>> "track_on_alarm", connectionProps)
>>
>> HTH.
>>
>> -Todd
>>
>> On Fri, Jan 8, 2016 at 10:53 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Which Spark release are you using ?
>>>
>>> For case #2, was there any error / clue in the logs ?
>>>
>>> Cheers
>>>
>>> On Fri, Jan 8, 2016 at 7:36 AM, Yasemin Kaya <go...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I want to write dataframe existing mysql table, but when i use
>>>> *peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>>> "track_on_alarm",false)*
>>>>
>>>> it says "Table track_on_alarm already exists."
>>>>
>>>> And when i *use peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>>> "track_on_alarm",true)*
>>>>
>>>> i lost the existing data.
>>>>
>>>> How i can write new data to db?
>>>>
>>>> Best,
>>>> yasemin
>>>>
>>>> --
>>>> hiç ender hiç
>>>>
>>>
>>>
>>
>
>
> --
> hiç ender hiç
>
Re: write new data to mysql
Posted by Yasemin Kaya <go...@gmail.com>.
When i change the version to 1.6.0, it worked.
Thanks.
2016-01-08 21:27 GMT+02:00 Yasemin Kaya <go...@gmail.com>:
> Hi,
> There is no write function that Todd mentioned or i cant find it.
> The code and error are in gist
> <https://gist.github.com/yaseminn/f5a2b78b126df71dfd0b>. Could you check
> it out please?
>
> Best,
> yasemin
>
> 2016-01-08 18:23 GMT+02:00 Todd Nist <ts...@gmail.com>:
>
>> It is not clear from the information provided why the insertIntoJDBC
>> failed in #2. I would note that method on the DataFrame as been deprecated
>> since 1.4, not sure what version your on. You should be able to do
>> something like this:
>>
>> DataFrame.write.mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL_WRITE,
>> "track_on_alarm", connectionProps)
>>
>> HTH.
>>
>> -Todd
>>
>> On Fri, Jan 8, 2016 at 10:53 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Which Spark release are you using ?
>>>
>>> For case #2, was there any error / clue in the logs ?
>>>
>>> Cheers
>>>
>>> On Fri, Jan 8, 2016 at 7:36 AM, Yasemin Kaya <go...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I want to write dataframe existing mysql table, but when i use
>>>> *peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>>> "track_on_alarm",false)*
>>>>
>>>> it says "Table track_on_alarm already exists."
>>>>
>>>> And when i *use peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>>> "track_on_alarm",true)*
>>>>
>>>> i lost the existing data.
>>>>
>>>> How i can write new data to db?
>>>>
>>>> Best,
>>>> yasemin
>>>>
>>>> --
>>>> hiç ender hiç
>>>>
>>>
>>>
>>
>
>
> --
> hiç ender hiç
>
--
hiç ender hiç
Re: write new data to mysql
Posted by Yasemin Kaya <go...@gmail.com>.
Hi,
There is no write function that Todd mentioned or i cant find it.
The code and error are in gist
<https://gist.github.com/yaseminn/f5a2b78b126df71dfd0b>. Could you check it
out please?
Best,
yasemin
2016-01-08 18:23 GMT+02:00 Todd Nist <ts...@gmail.com>:
> It is not clear from the information provided why the insertIntoJDBC
> failed in #2. I would note that method on the DataFrame as been deprecated
> since 1.4, not sure what version your on. You should be able to do
> something like this:
>
> DataFrame.write.mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL_WRITE,
> "track_on_alarm", connectionProps)
>
> HTH.
>
> -Todd
>
> On Fri, Jan 8, 2016 at 10:53 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Which Spark release are you using ?
>>
>> For case #2, was there any error / clue in the logs ?
>>
>> Cheers
>>
>> On Fri, Jan 8, 2016 at 7:36 AM, Yasemin Kaya <go...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I want to write dataframe existing mysql table, but when i use
>>> *peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>> "track_on_alarm",false)*
>>>
>>> it says "Table track_on_alarm already exists."
>>>
>>> And when i *use peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>>> "track_on_alarm",true)*
>>>
>>> i lost the existing data.
>>>
>>> How i can write new data to db?
>>>
>>> Best,
>>> yasemin
>>>
>>> --
>>> hiç ender hiç
>>>
>>
>>
>
--
hiç ender hiç
Re: write new data to mysql
Posted by Todd Nist <ts...@gmail.com>.
It is not clear from the information provided why the insertIntoJDBC failed
in #2. I would note that method on the DataFrame as been deprecated since
1.4, not sure what version your on. You should be able to do something
like this:
DataFrame.write.mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL_WRITE,
"track_on_alarm", connectionProps)
HTH.
-Todd
On Fri, Jan 8, 2016 at 10:53 AM, Ted Yu <yu...@gmail.com> wrote:
> Which Spark release are you using ?
>
> For case #2, was there any error / clue in the logs ?
>
> Cheers
>
> On Fri, Jan 8, 2016 at 7:36 AM, Yasemin Kaya <go...@gmail.com> wrote:
>
>> Hi,
>>
>> I want to write dataframe existing mysql table, but when i use
>> *peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>> "track_on_alarm",false)*
>>
>> it says "Table track_on_alarm already exists."
>>
>> And when i *use peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
>> "track_on_alarm",true)*
>>
>> i lost the existing data.
>>
>> How i can write new data to db?
>>
>> Best,
>> yasemin
>>
>> --
>> hiç ender hiç
>>
>
>
Re: write new data to mysql
Posted by Ted Yu <yu...@gmail.com>.
Which Spark release are you using ?
For case #2, was there any error / clue in the logs ?
Cheers
On Fri, Jan 8, 2016 at 7:36 AM, Yasemin Kaya <go...@gmail.com> wrote:
> Hi,
>
> I want to write dataframe existing mysql table, but when i use
> *peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
> "track_on_alarm",false)*
>
> it says "Table track_on_alarm already exists."
>
> And when i *use peopleDataFrame.insertIntoJDBC(MYSQL_CONNECTION_URL_WRITE,
> "track_on_alarm",true)*
>
> i lost the existing data.
>
> How i can write new data to db?
>
> Best,
> yasemin
>
> --
> hiç ender hiç
>