You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andrés Ivaldi <ia...@gmail.com> on 2016/04/29 17:16:06 UTC
Bit(N) on create Table with MSSQLServer
Hello, Spark is executing a create table sentence (using JDBC) to
MSSQLServer with a mapping column type like ColName Bit(1) for boolean
types, This create table cannot be executed on MSSQLServer.
In class JdbcDialect the mapping for Boolean type is Bit(1), so the
question is, this is a problem of spark or JDBC driver who is not mapping
right?
Anyway it´s possible to override that mapping in Spark?
Regards
--
Ing. Ivaldi Andres
Re: Bit(N) on create Table with MSSQLServer
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hang on
Are you talking about target database in MSSQL created and dropped?
Is your Spark JDBC credential in MSSQL a DBO or sa?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 4 May 2016 at 20:54, Andrés Ivaldi <ia...@gmail.com> wrote:
> Ok, so I did that, create a database and then insert data, but spark drop
> database and try to create it again, I'm using
> Dataframe.write(SaveMode.Overwrite), documentation said :
> "when performing a Overwrite, the data will be deleted before writing out
> the new data."
>
> why is dropping the table?
>
>
> On Wed, May 4, 2016 at 6:44 AM, Andrés Ivaldi <ia...@gmail.com> wrote:
>
>> Yes, I can do that, it's what we are doing now, but I think the best
>> approach would be delegate the create table action to spark.
>>
>> On Tue, May 3, 2016 at 8:17 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Can you create the MSSQL (target) table first with the correct column
>>> setting and insert data from Spark to it with JDBC as opposed to JDBC
>>> creating target table itself?
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 3 May 2016 at 22:19, Andrés Ivaldi <ia...@gmail.com> wrote:
>>>
>>>> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is
>>>> Text instead of varchar(MAX) , so how can I override default SQL Mapping?
>>>>
>>>> regards.
>>>>
>>>> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> Well if MSSQL cannot create that column then it is more like
>>>>> compatibility between Spark and RDBMS.
>>>>>
>>>>> What value that column has in MSSQL. Can you create table the table in
>>>>> MSSQL database or map it in Spark to a valid column before opening JDBC
>>>>> connection?
>>>>>
>>>>> HTH
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>>
>>>>> On 29 April 2016 at 16:16, Andrés Ivaldi <ia...@gmail.com> wrote:
>>>>>
>>>>>> Hello, Spark is executing a create table sentence (using JDBC) to
>>>>>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>>>>>> types, This create table cannot be executed on MSSQLServer.
>>>>>>
>>>>>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>>>>>> question is, this is a problem of spark or JDBC driver who is not mapping
>>>>>> right?
>>>>>>
>>>>>> Anyway it´s possible to override that mapping in Spark?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> --
>>>>>> Ing. Ivaldi Andres
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Ing. Ivaldi Andres
>>>>
>>>
>>>
>>
>>
>> --
>> Ing. Ivaldi Andres
>>
>
>
>
> --
> Ing. Ivaldi Andres
>
Re: Bit(N) on create Table with MSSQLServer
Posted by Andrés Ivaldi <ia...@gmail.com>.
Ok, so I did that, create a database and then insert data, but spark drop
database and try to create it again, I'm using
Dataframe.write(SaveMode.Overwrite), documentation said :
"when performing a Overwrite, the data will be deleted before writing out
the new data."
why is dropping the table?
On Wed, May 4, 2016 at 6:44 AM, Andrés Ivaldi <ia...@gmail.com> wrote:
> Yes, I can do that, it's what we are doing now, but I think the best
> approach would be delegate the create table action to spark.
>
> On Tue, May 3, 2016 at 8:17 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> Can you create the MSSQL (target) table first with the correct column
>> setting and insert data from Spark to it with JDBC as opposed to JDBC
>> creating target table itself?
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 3 May 2016 at 22:19, Andrés Ivaldi <ia...@gmail.com> wrote:
>>
>>> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
>>> instead of varchar(MAX) , so how can I override default SQL Mapping?
>>>
>>> regards.
>>>
>>> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com> wrote:
>>>
>>>> Well if MSSQL cannot create that column then it is more like
>>>> compatibility between Spark and RDBMS.
>>>>
>>>> What value that column has in MSSQL. Can you create table the table in
>>>> MSSQL database or map it in Spark to a valid column before opening JDBC
>>>> connection?
>>>>
>>>> HTH
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>> On 29 April 2016 at 16:16, Andrés Ivaldi <ia...@gmail.com> wrote:
>>>>
>>>>> Hello, Spark is executing a create table sentence (using JDBC) to
>>>>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>>>>> types, This create table cannot be executed on MSSQLServer.
>>>>>
>>>>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>>>>> question is, this is a problem of spark or JDBC driver who is not mapping
>>>>> right?
>>>>>
>>>>> Anyway it´s possible to override that mapping in Spark?
>>>>>
>>>>> Regards
>>>>>
>>>>> --
>>>>> Ing. Ivaldi Andres
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Ing. Ivaldi Andres
>>>
>>
>>
>
>
> --
> Ing. Ivaldi Andres
>
--
Ing. Ivaldi Andres
Re: Bit(N) on create Table with MSSQLServer
Posted by Andrés Ivaldi <ia...@gmail.com>.
Yes, I can do that, it's what we are doing now, but I think the best
approach would be delegate the create table action to spark.
On Tue, May 3, 2016 at 8:17 PM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Can you create the MSSQL (target) table first with the correct column
> setting and insert data from Spark to it with JDBC as opposed to JDBC
> creating target table itself?
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 3 May 2016 at 22:19, Andrés Ivaldi <ia...@gmail.com> wrote:
>
>> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
>> instead of varchar(MAX) , so how can I override default SQL Mapping?
>>
>> regards.
>>
>> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Well if MSSQL cannot create that column then it is more like
>>> compatibility between Spark and RDBMS.
>>>
>>> What value that column has in MSSQL. Can you create table the table in
>>> MSSQL database or map it in Spark to a valid column before opening JDBC
>>> connection?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 29 April 2016 at 16:16, Andrés Ivaldi <ia...@gmail.com> wrote:
>>>
>>>> Hello, Spark is executing a create table sentence (using JDBC) to
>>>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>>>> types, This create table cannot be executed on MSSQLServer.
>>>>
>>>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>>>> question is, this is a problem of spark or JDBC driver who is not mapping
>>>> right?
>>>>
>>>> Anyway it´s possible to override that mapping in Spark?
>>>>
>>>> Regards
>>>>
>>>> --
>>>> Ing. Ivaldi Andres
>>>>
>>>
>>>
>>
>>
>> --
>> Ing. Ivaldi Andres
>>
>
>
--
Ing. Ivaldi Andres
Re: Bit(N) on create Table with MSSQLServer
Posted by Mich Talebzadeh <mi...@gmail.com>.
Can you create the MSSQL (target) table first with the correct column
setting and insert data from Spark to it with JDBC as opposed to JDBC
creating target table itself?
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 3 May 2016 at 22:19, Andrés Ivaldi <ia...@gmail.com> wrote:
> Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
> instead of varchar(MAX) , so how can I override default SQL Mapping?
>
> regards.
>
> On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> Well if MSSQL cannot create that column then it is more like
>> compatibility between Spark and RDBMS.
>>
>> What value that column has in MSSQL. Can you create table the table in
>> MSSQL database or map it in Spark to a valid column before opening JDBC
>> connection?
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 29 April 2016 at 16:16, Andrés Ivaldi <ia...@gmail.com> wrote:
>>
>>> Hello, Spark is executing a create table sentence (using JDBC) to
>>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>>> types, This create table cannot be executed on MSSQLServer.
>>>
>>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>>> question is, this is a problem of spark or JDBC driver who is not mapping
>>> right?
>>>
>>> Anyway it´s possible to override that mapping in Spark?
>>>
>>> Regards
>>>
>>> --
>>> Ing. Ivaldi Andres
>>>
>>
>>
>
>
> --
> Ing. Ivaldi Andres
>
Re: Bit(N) on create Table with MSSQLServer
Posted by Andrés Ivaldi <ia...@gmail.com>.
Ok, Spark MSSQL dataType mapping is not right for me, ie. string is Text
instead of varchar(MAX) , so how can I override default SQL Mapping?
regards.
On Sun, May 1, 2016 at 5:23 AM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Well if MSSQL cannot create that column then it is more like compatibility
> between Spark and RDBMS.
>
> What value that column has in MSSQL. Can you create table the table in
> MSSQL database or map it in Spark to a valid column before opening JDBC
> connection?
>
> HTH
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 29 April 2016 at 16:16, Andrés Ivaldi <ia...@gmail.com> wrote:
>
>> Hello, Spark is executing a create table sentence (using JDBC) to
>> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
>> types, This create table cannot be executed on MSSQLServer.
>>
>> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
>> question is, this is a problem of spark or JDBC driver who is not mapping
>> right?
>>
>> Anyway it´s possible to override that mapping in Spark?
>>
>> Regards
>>
>> --
>> Ing. Ivaldi Andres
>>
>
>
--
Ing. Ivaldi Andres
Re: Bit(N) on create Table with MSSQLServer
Posted by Mich Talebzadeh <mi...@gmail.com>.
Well if MSSQL cannot create that column then it is more like compatibility
between Spark and RDBMS.
What value that column has in MSSQL. Can you create table the table in
MSSQL database or map it in Spark to a valid column before opening JDBC
connection?
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 29 April 2016 at 16:16, Andrés Ivaldi <ia...@gmail.com> wrote:
> Hello, Spark is executing a create table sentence (using JDBC) to
> MSSQLServer with a mapping column type like ColName Bit(1) for boolean
> types, This create table cannot be executed on MSSQLServer.
>
> In class JdbcDialect the mapping for Boolean type is Bit(1), so the
> question is, this is a problem of spark or JDBC driver who is not mapping
> right?
>
> Anyway it´s possible to override that mapping in Spark?
>
> Regards
>
> --
> Ing. Ivaldi Andres
>