You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aviral Agarwal <av...@gmail.com> on 2017/06/21 09:06:59 UTC

JDBC RDD Timestamp Parsing Issue

Hi,

I am using JDBC RDD to read from a MySQL RDBMS.
My spark job fails with the below error :

java.sql.SQLException: Value '0000-00-00 00:00:00.000' can not be
represented as java.sql.Timestamp


Now instead of the whole job failing I want to skip this record and
continue processing the rest.
Any leads on how that can be done ?


Thanks and Regards,
Aviral Agarwal

Re: JDBC RDD Timestamp Parsing Issue

Posted by Aviral Agarwal <av...@gmail.com>.
This woks. Thanks !

- Aviral Agarwal

On Wed, Jun 21, 2017 at 6:07 PM, Eduardo Mello <ee...@gmail.com> wrote:

> You can add "?zeroDateTimeBehavior=convertToNull" to the connection
> string.
>
> On Wed, Jun 21, 2017 at 9:04 AM, Aviral Agarwal <av...@gmail.com>
> wrote:
>
>> The exception is happening in JDBC RDD code where getNext() is called to
>> get the next row.
>> I do not have access to the result set. I am operating on a DataFrame.
>>
>> Thanks and Regards,
>> Aviral Agarwal
>>
>> On Jun 21, 2017 17:19, "Mahesh Sawaiker" <ma...@persistent.com>
>> wrote:
>>
>>> This has to do with how you are creating the timestamp object from the
>>> resultset ( I guess).
>>>
>>> If you can provide more code it will help, but you could surround the
>>> parsing code with a try catch and then just ignore the exception.
>>>
>>>
>>>
>>> *From:* Aviral Agarwal [mailto:aviral12028@gmail.com]
>>> *Sent:* Wednesday, June 21, 2017 2:37 PM
>>> *To:* user@spark.apache.org
>>> *Subject:* JDBC RDD Timestamp Parsing Issue
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> I am using JDBC RDD to read from a MySQL RDBMS.
>>>
>>> My spark job fails with the below error :
>>>
>>>
>>>
>>> java.sql.SQLException: Value '0000-00-00 00:00:00.000' can not be represented as java.sql.Timestamp
>>>
>>>
>>>
>>> Now instead of the whole job failing I want to skip this record and
>>> continue processing the rest.
>>> Any leads on how that can be done ?
>>>
>>>
>>> Thanks and Regards,
>>> Aviral Agarwal
>>> DISCLAIMER
>>> ==========
>>> This e-mail may contain privileged and confidential information which is
>>> the property of Persistent Systems Ltd. It is intended only for the use of
>>> the individual or entity to which it is addressed. If you are not the
>>> intended recipient, you are not authorized to read, retain, copy, print,
>>> distribute or use this message. If you have received this communication in
>>> error, please notify the sender and delete all copies of this message.
>>> Persistent Systems Ltd. does not accept any liability for virus infected
>>> mails.
>>>
>>
>

Re: JDBC RDD Timestamp Parsing Issue

Posted by Eduardo Mello <ee...@gmail.com>.
You can add "?zeroDateTimeBehavior=convertToNull" to the connection string.

On Wed, Jun 21, 2017 at 9:04 AM, Aviral Agarwal <av...@gmail.com>
wrote:

> The exception is happening in JDBC RDD code where getNext() is called to
> get the next row.
> I do not have access to the result set. I am operating on a DataFrame.
>
> Thanks and Regards,
> Aviral Agarwal
>
> On Jun 21, 2017 17:19, "Mahesh Sawaiker" <ma...@persistent.com>
> wrote:
>
>> This has to do with how you are creating the timestamp object from the
>> resultset ( I guess).
>>
>> If you can provide more code it will help, but you could surround the
>> parsing code with a try catch and then just ignore the exception.
>>
>>
>>
>> *From:* Aviral Agarwal [mailto:aviral12028@gmail.com]
>> *Sent:* Wednesday, June 21, 2017 2:37 PM
>> *To:* user@spark.apache.org
>> *Subject:* JDBC RDD Timestamp Parsing Issue
>>
>>
>>
>> Hi,
>>
>>
>>
>> I am using JDBC RDD to read from a MySQL RDBMS.
>>
>> My spark job fails with the below error :
>>
>>
>>
>> java.sql.SQLException: Value '0000-00-00 00:00:00.000' can not be represented as java.sql.Timestamp
>>
>>
>>
>> Now instead of the whole job failing I want to skip this record and
>> continue processing the rest.
>> Any leads on how that can be done ?
>>
>>
>> Thanks and Regards,
>> Aviral Agarwal
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>

RE: JDBC RDD Timestamp Parsing Issue

Posted by Aviral Agarwal <av...@gmail.com>.
The exception is happening in JDBC RDD code where getNext() is called to
get the next row.
I do not have access to the result set. I am operating on a DataFrame.

Thanks and Regards,
Aviral Agarwal

On Jun 21, 2017 17:19, "Mahesh Sawaiker" <ma...@persistent.com>
wrote:

> This has to do with how you are creating the timestamp object from the
> resultset ( I guess).
>
> If you can provide more code it will help, but you could surround the
> parsing code with a try catch and then just ignore the exception.
>
>
>
> *From:* Aviral Agarwal [mailto:aviral12028@gmail.com]
> *Sent:* Wednesday, June 21, 2017 2:37 PM
> *To:* user@spark.apache.org
> *Subject:* JDBC RDD Timestamp Parsing Issue
>
>
>
> Hi,
>
>
>
> I am using JDBC RDD to read from a MySQL RDBMS.
>
> My spark job fails with the below error :
>
>
>
> java.sql.SQLException: Value '0000-00-00 00:00:00.000' can not be represented as java.sql.Timestamp
>
>
>
> Now instead of the whole job failing I want to skip this record and
> continue processing the rest.
> Any leads on how that can be done ?
>
>
> Thanks and Regards,
> Aviral Agarwal
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>

RE: JDBC RDD Timestamp Parsing Issue

Posted by Mahesh Sawaiker <ma...@persistent.com>.
This has to do with how you are creating the timestamp object from the resultset ( I guess).
If you can provide more code it will help, but you could surround the parsing code with a try catch and then just ignore the exception.

From: Aviral Agarwal [mailto:aviral12028@gmail.com]
Sent: Wednesday, June 21, 2017 2:37 PM
To: user@spark.apache.org
Subject: JDBC RDD Timestamp Parsing Issue

Hi,

I am using JDBC RDD to read from a MySQL RDBMS.
My spark job fails with the below error :


java.sql.SQLException: Value '0000-00-00 00:00:00.000' can not be represented as java.sql.Timestamp


Now instead of the whole job failing I want to skip this record and continue processing the rest.
Any leads on how that can be done ?


Thanks and Regards,
Aviral Agarwal
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.