You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mahender Sarangam <Ma...@outlook.com> on 2016/07/18 20:59:54 UTC
Re: ORC does not support type conversion from INT to STRING.
Hi Mich,
Sorry for delay in responding. here is the scenario,
We have created new cluster and we have moved all ORC File data into new cluster. We have re-created table pointing to ORC location. We have modified data type of ORC table from INT to String. From then onward, we were unable to fire select statement against this ORC table, hive keep throwing exception, "Orc table select. Unable to convert Int to String". Looks like it is bug in ORC table only. Where in we modify the datatype from int to string, is causing problem with ORC reading/select statement, it throws exceptio. Please let me know if there are any workaround for this scenario. Is this behavior expected previously also.
/Mahender
On 6/14/2016 11:47 AM, Mich Talebzadeh wrote:
you must excuse my ignorance
can you please elaborate on this as there seems something has gone wrong somewhere?
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>
On 14 June 2016 at 19:42, Mahender Sarangam <Ma...@outlook.com>> wrote:
Yes Mich. We have restored cluster from metastore.
On 6/14/2016 11:35 AM, Mich Talebzadeh wrote:
Hi Mahendar,
Did you load the meta-data DB/schema from backup and now seeing this error
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
On 14 June 2016 at 19:04, Mahender Sarangam <Ma...@outlook.com>> wrote:
ping.
On 6/13/2016 1:19 PM, Mahender Sarangam wrote:
Hi,
We are facing issue while reading data from ORC table. We have created ORC table and dumped data into it. We have deleted cluster due to some reason. When we recreated cluster (using Metastore) and table pointing to same location. When we perform reading from ORC table. We see below error.
SELECT col2, Col1,
reflect("java.util.UUID", "randomUUID") AS ID,
Source,
1 ,
SDate,
EDate
FROM Table ORC JOIN Table2 _surr;
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1465411930667_0212_1_01, diagnostics=[Task failed, taskId=task_1465411930667_0212_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: ORC does not support type conversion from INT to STRING.
I think issue is reflect("java.util.UUID", "randomUUID") AS ID
I know there is Bug raised while reading data from ORC table. Is there any workaround apart from reloading data.
-MS
Re: ORC does not support type conversion from INT to STRING.
Posted by Mich Talebzadeh <mi...@gmail.com>.
Is that a distro from Hortonworks? In that case what Matthew mentioned may
be valid. Unless you go through pain of inserting using CAST function?
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 19 July 2016 at 22:19, Mahender Sarangam <Ma...@outlook.com>
wrote:
> But we are using Hive 1.2 version
>
>
> On 7/19/2016 12:43 PM, Mich Talebzadeh wrote:
>
> in Hive 2, I don't see this issue INSERT/SELECT from INT to String column!
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 19 July 2016 at 20:39, Mahender Sarangam <Ma...@outlook.com>
> wrote:
>
>>
>> Thanks Matthew,
>>
>> Currently we are in Hive 1.2 version only, Is there any setting like
>> "hive.metastore.disallow.incompatible.col.type.changes=false;" in Hive 1.2
>> or any around apart for reloading entire table data. For Quick workaround,
>> we are reloading entire data.
>> Can you please share with us Jira for Schema Evolution.
>>
>>
>> @Mich : Currently we have only primitive types. But I'm also interested
>> to know "how the behavior will be in complex types"
>>
>>
>> /Mahender
>>
>>
>> On 7/18/2016 3:55 PM, Mich Talebzadeh wrote:
>>
>> Hi Mathew,
>>
>> In layman's term if I create the source ORC table column as INT and then
>> create a target ORC table but that column has now been defined as STRING
>> and do an INSERT/SELECT from source table how data is internally stored?
>>
>> Is it implicitly converted into new format using CAST function or it is
>> stored as is and just masked?
>>
>> The version of Hive I am using is 2 and it works OK for primitive data
>> types (insert/select from INT to String)
>>
>> However, I believe Mahender is referring to Complex types?
>>
>> Thanks
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 18 July 2016 at 22:31, Matthew McCline <mm...@hortonworks.com>
>> wrote:
>>
>>>
>>> Hi Mahender,
>>>
>>>
>>> Schema Evolution is available on the latest recent version of Hive.
>>>
>>>
>>> For example, if you set
>>> hive.metastore.disallow.incompatible.col.type.changes=false; on master
>>> (i.e. hive2) it will support INT to STRING conversion.
>>>
>>>
>>> If you need to remain on an older version, then you are out of luck.
>>>
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>> ------------------------------
>>> *From:* Mahender Sarangam <Ma...@outlook.com>
>>> *Sent:* Monday, July 18, 2016 1:59 PM
>>> *To:* user@hive.apache.org
>>> *Subject:* Re: ORC does not support type conversion from INT to STRING.
>>>
>>>
>>> Hi Mich,
>>>
>>> Sorry for delay in responding. here is the scenario,
>>>
>>> We have created new cluster and we have moved all ORC File data into
>>> new cluster. We have re-created table pointing to ORC location. We have
>>> modified data type of ORC table from *INT *to *String.* From then
>>> onward, we were unable to fire select statement against this ORC table,
>>> hive keep throwing exception, "Orc table select. Unable to convert Int to
>>> String". Looks like it is bug in ORC table only. Where in we modify the
>>> datatype from *int to string,* is causing problem with ORC
>>> reading/select statement, it throws exceptio. Please let me know if there
>>> are any workaround for this scenario. Is this behavior expected previously
>>> also.
>>>
>>>
>>> */Mahender*
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 6/14/2016 11:47 AM, Mich Talebzadeh wrote:
>>>
>>> you must excuse my ignorance
>>>
>>> can you please elaborate on this as there seems something has gone wrong
>>> somewhere?
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 14 June 2016 at 19:42, Mahender Sarangam <
>>> Mahender.BigData@outlook.com> wrote:
>>>
>>>> Yes Mich. We have restored cluster from metastore.
>>>>
>>>> On 6/14/2016 11:35 AM, Mich Talebzadeh wrote:
>>>>
>>>> Hi Mahendar,
>>>>
>>>>
>>>> Did you load the meta-data DB/schema from backup and now seeing this
>>>> error
>>>>
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>>
>>>> On 14 June 2016 at 19:04, Mahender Sarangam <
>>>> Mahender.BigData@outlook.com> wrote:
>>>>
>>>>> ping.
>>>>>
>>>>> On 6/13/2016 1:19 PM, Mahender Sarangam wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We are facing issue while reading data from ORC table. We have created
>>>>> ORC table and dumped data into it. We have deleted cluster due to some
>>>>> reason. When we recreated cluster (using Metastore) and table pointing to
>>>>> same location. When we perform reading from ORC table. We see below error.
>>>>>
>>>>> SELECT col2, Col1,
>>>>> reflect("java.util.UUID", "randomUUID") AS ID,
>>>>> Source,
>>>>> 1 ,
>>>>> SDate,
>>>>> EDate
>>>>> FROM Table ORC JOIN Table2 _surr;
>>>>>
>>>>> ERROR : Vertex failed, vertexName=Map 1,
>>>>> vertexId=vertex_1465411930667_0212_1_01, diagnostics=[Task failed,
>>>>> taskId=task_1465411930667_0212_1_01_000000, diagnostics=[TaskAttempt 0
>>>>> failed, info=[Error: Failure while running task:java.lang.RuntimeException:
>>>>> java.lang.RuntimeException: java.io.IOException: java.io.IOException: ORC
>>>>> does not support type conversion from INT to STRING.
>>>>>
>>>>>
>>>>> I think issue is reflect("java.util.UUID", "randomUUID") AS ID
>>>>>
>>>>>
>>>>> I know there is Bug raised while reading data from ORC table. Is there
>>>>> any workaround apart from reloading data.
>>>>>
>>>>> -MS
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: ORC does not support type conversion from INT to STRING.
Posted by Mahender Sarangam <Ma...@outlook.com>.
But we are using Hive 1.2 version
On 7/19/2016 12:43 PM, Mich Talebzadeh wrote:
in Hive 2, I don't see this issue INSERT/SELECT from INT to String column!
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 19 July 2016 at 20:39, Mahender Sarangam <Ma...@outlook.com>> wrote:
Thanks Matthew,
Currently we are in Hive 1.2 version only, Is there any setting like "hive.metastore.disallow.incompatible.col.type.changes=false;" in Hive 1.2 or any around apart for reloading entire table data. For Quick workaround, we are reloading entire data.
Can you please share with us Jira for Schema Evolution.
@Mich : Currently we have only primitive types. But I'm also interested to know "how the behavior will be in complex types"
/Mahender
On 7/18/2016 3:55 PM, Mich Talebzadeh wrote:
Hi Mathew,
In layman's term if I create the source ORC table column as INT and then create a target ORC table but that column has now been defined as STRING and do an INSERT/SELECT from source table how data is internally stored?
Is it implicitly converted into new format using CAST function or it is stored as is and just masked?
The version of Hive I am using is 2 and it works OK for primitive data types (insert/select from INT to String)
However, I believe Mahender is referring to Complex types?
Thanks
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 18 July 2016 at 22:31, Matthew McCline <mm...@hortonworks.com>> wrote:
Hi Mahender,
Schema Evolution is available on the latest recent version of Hive.
For example, if you set hive.metastore.disallow.incompatible.col.type.changes=false; on master (i.e. hive2) it will support INT to STRING conversion.
If you need to remain on an older version, then you are out of luck.
Thanks,
Matt
________________________________
From: Mahender Sarangam <Ma...@outlook.com>>
Sent: Monday, July 18, 2016 1:59 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: ORC does not support type conversion from INT to STRING.
Hi Mich,
Sorry for delay in responding. here is the scenario,
We have created new cluster and we have moved all ORC File data into new cluster. We have re-created table pointing to ORC location. We have modified data type of ORC table from INT to String. From then onward, we were unable to fire select statement against this ORC table, hive keep throwing exception, "Orc table select. Unable to convert Int to String". Looks like it is bug in ORC table only. Where in we modify the datatype from int to string, is causing problem with ORC reading/select statement, it throws exceptio. Please let me know if there are any workaround for this scenario. Is this behavior expected previously also.
/Mahender
On 6/14/2016 11:47 AM, Mich Talebzadeh wrote:
you must excuse my ignorance
can you please elaborate on this as there seems something has gone wrong somewhere?
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>
On 14 June 2016 at 19:42, Mahender Sarangam <Ma...@outlook.com>> wrote:
Yes Mich. We have restored cluster from metastore.
On 6/14/2016 11:35 AM, Mich Talebzadeh wrote:
Hi Mahendar,
Did you load the meta-data DB/schema from backup and now seeing this error
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
On 14 June 2016 at 19:04, Mahender Sarangam <Ma...@outlook.com>> wrote:
ping.
On 6/13/2016 1:19 PM, Mahender Sarangam wrote:
Hi,
We are facing issue while reading data from ORC table. We have created ORC table and dumped data into it. We have deleted cluster due to some reason. When we recreated cluster (using Metastore) and table pointing to same location. When we perform reading from ORC table. We see below error.
SELECT col2, Col1,
reflect("java.util.UUID", "randomUUID") AS ID,
Source,
1 ,
SDate,
EDate
FROM Table ORC JOIN Table2 _surr;
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1465411930667_0212_1_01, diagnostics=[Task failed, taskId=task_1465411930667_0212_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: ORC does not support type conversion from INT to STRING.
I think issue is reflect("java.util.UUID", "randomUUID") AS ID
I know there is Bug raised while reading data from ORC table. Is there any workaround apart from reloading data.
-MS
Re: ORC does not support type conversion from INT to STRING.
Posted by Mich Talebzadeh <mi...@gmail.com>.
in Hive 2, I don't see this issue INSERT/SELECT from INT to String column!
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 19 July 2016 at 20:39, Mahender Sarangam <Ma...@outlook.com>
wrote:
>
> Thanks Matthew,
>
> Currently we are in Hive 1.2 version only, Is there any setting like
> "hive.metastore.disallow.incompatible.col.type.changes=false;" in Hive 1.2
> or any around apart for reloading entire table data. For Quick workaround,
> we are reloading entire data.
> Can you please share with us Jira for Schema Evolution.
>
>
> @Mich : Currently we have only primitive types. But I'm also interested to
> know "how the behavior will be in complex types"
>
>
> /Mahender
>
>
> On 7/18/2016 3:55 PM, Mich Talebzadeh wrote:
>
> Hi Mathew,
>
> In layman's term if I create the source ORC table column as INT and then
> create a target ORC table but that column has now been defined as STRING
> and do an INSERT/SELECT from source table how data is internally stored?
>
> Is it implicitly converted into new format using CAST function or it is
> stored as is and just masked?
>
> The version of Hive I am using is 2 and it works OK for primitive data
> types (insert/select from INT to String)
>
> However, I believe Mahender is referring to Complex types?
>
> Thanks
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 18 July 2016 at 22:31, Matthew McCline <mm...@hortonworks.com>
> wrote:
>
>>
>> Hi Mahender,
>>
>>
>> Schema Evolution is available on the latest recent version of Hive.
>>
>>
>> For example, if you set
>> hive.metastore.disallow.incompatible.col.type.changes=false; on master
>> (i.e. hive2) it will support INT to STRING conversion.
>>
>>
>> If you need to remain on an older version, then you are out of luck.
>>
>>
>> Thanks,
>>
>> Matt
>>
>>
>> ------------------------------
>> *From:* Mahender Sarangam <Ma...@outlook.com>
>> *Sent:* Monday, July 18, 2016 1:59 PM
>> *To:* user@hive.apache.org
>> *Subject:* Re: ORC does not support type conversion from INT to STRING.
>>
>>
>> Hi Mich,
>>
>> Sorry for delay in responding. here is the scenario,
>>
>> We have created new cluster and we have moved all ORC File data into new
>> cluster. We have re-created table pointing to ORC location. We have
>> modified data type of ORC table from *INT *to *String.* From then
>> onward, we were unable to fire select statement against this ORC table,
>> hive keep throwing exception, "Orc table select. Unable to convert Int to
>> String". Looks like it is bug in ORC table only. Where in we modify the
>> datatype from *int to string,* is causing problem with ORC
>> reading/select statement, it throws exceptio. Please let me know if there
>> are any workaround for this scenario. Is this behavior expected previously
>> also.
>>
>>
>> */Mahender*
>>
>>
>>
>>
>>
>>
>> On 6/14/2016 11:47 AM, Mich Talebzadeh wrote:
>>
>> you must excuse my ignorance
>>
>> can you please elaborate on this as there seems something has gone wrong
>> somewhere?
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 14 June 2016 at 19:42, Mahender Sarangam <Mahender.BigData@outlook.com
>> > wrote:
>>
>>> Yes Mich. We have restored cluster from metastore.
>>>
>>> On 6/14/2016 11:35 AM, Mich Talebzadeh wrote:
>>>
>>> Hi Mahendar,
>>>
>>>
>>> Did you load the meta-data DB/schema from backup and now seeing this
>>> error
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 14 June 2016 at 19:04, Mahender Sarangam <
>>> Mahender.BigData@outlook.com> wrote:
>>>
>>>> ping.
>>>>
>>>> On 6/13/2016 1:19 PM, Mahender Sarangam wrote:
>>>>
>>>> Hi,
>>>>
>>>> We are facing issue while reading data from ORC table. We have created
>>>> ORC table and dumped data into it. We have deleted cluster due to some
>>>> reason. When we recreated cluster (using Metastore) and table pointing to
>>>> same location. When we perform reading from ORC table. We see below error.
>>>>
>>>> SELECT col2, Col1,
>>>> reflect("java.util.UUID", "randomUUID") AS ID,
>>>> Source,
>>>> 1 ,
>>>> SDate,
>>>> EDate
>>>> FROM Table ORC JOIN Table2 _surr;
>>>>
>>>> ERROR : Vertex failed, vertexName=Map 1,
>>>> vertexId=vertex_1465411930667_0212_1_01, diagnostics=[Task failed,
>>>> taskId=task_1465411930667_0212_1_01_000000, diagnostics=[TaskAttempt 0
>>>> failed, info=[Error: Failure while running task:java.lang.RuntimeException:
>>>> java.lang.RuntimeException: java.io.IOException: java.io.IOException: ORC
>>>> does not support type conversion from INT to STRING.
>>>>
>>>>
>>>> I think issue is reflect("java.util.UUID", "randomUUID") AS ID
>>>>
>>>>
>>>> I know there is Bug raised while reading data from ORC table. Is there
>>>> any workaround apart from reloading data.
>>>>
>>>> -MS
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
Re: ORC does not support type conversion from INT to STRING.
Posted by Mahender Sarangam <Ma...@outlook.com>.
Thanks Matthew,
Currently we are in Hive 1.2 version only, Is there any setting like "hive.metastore.disallow.incompatible.col.type.changes=false;" in Hive 1.2 or any around apart for reloading entire table data. For Quick workaround, we are reloading entire data.
Can you please share with us Jira for Schema Evolution.
@Mich : Currently we have only primitive types. But I'm also interested to know "how the behavior will be in complex types"
/Mahender
On 7/18/2016 3:55 PM, Mich Talebzadeh wrote:
Hi Mathew,
In layman's term if I create the source ORC table column as INT and then create a target ORC table but that column has now been defined as STRING and do an INSERT/SELECT from source table how data is internally stored?
Is it implicitly converted into new format using CAST function or it is stored as is and just masked?
The version of Hive I am using is 2 and it works OK for primitive data types (insert/select from INT to String)
However, I believe Mahender is referring to Complex types?
Thanks
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On 18 July 2016 at 22:31, Matthew McCline <mm...@hortonworks.com>> wrote:
Hi Mahender,
Schema Evolution is available on the latest recent version of Hive.
For example, if you set hive.metastore.disallow.incompatible.col.type.changes=false; on master (i.e. hive2) it will support INT to STRING conversion.
If you need to remain on an older version, then you are out of luck.
Thanks,
Matt
________________________________
From: Mahender Sarangam <Ma...@outlook.com>>
Sent: Monday, July 18, 2016 1:59 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: ORC does not support type conversion from INT to STRING.
Hi Mich,
Sorry for delay in responding. here is the scenario,
We have created new cluster and we have moved all ORC File data into new cluster. We have re-created table pointing to ORC location. We have modified data type of ORC table from INT to String. From then onward, we were unable to fire select statement against this ORC table, hive keep throwing exception, "Orc table select. Unable to convert Int to String". Looks like it is bug in ORC table only. Where in we modify the datatype from int to string, is causing problem with ORC reading/select statement, it throws exceptio. Please let me know if there are any workaround for this scenario. Is this behavior expected previously also.
/Mahender
On 6/14/2016 11:47 AM, Mich Talebzadeh wrote:
you must excuse my ignorance
can you please elaborate on this as there seems something has gone wrong somewhere?
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>
On 14 June 2016 at 19:42, Mahender Sarangam <Ma...@outlook.com>> wrote:
Yes Mich. We have restored cluster from metastore.
On 6/14/2016 11:35 AM, Mich Talebzadeh wrote:
Hi Mahendar,
Did you load the meta-data DB/schema from backup and now seeing this error
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
On 14 June 2016 at 19:04, Mahender Sarangam <Ma...@outlook.com>> wrote:
ping.
On 6/13/2016 1:19 PM, Mahender Sarangam wrote:
Hi,
We are facing issue while reading data from ORC table. We have created ORC table and dumped data into it. We have deleted cluster due to some reason. When we recreated cluster (using Metastore) and table pointing to same location. When we perform reading from ORC table. We see below error.
SELECT col2, Col1,
reflect("java.util.UUID", "randomUUID") AS ID,
Source,
1 ,
SDate,
EDate
FROM Table ORC JOIN Table2 _surr;
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1465411930667_0212_1_01, diagnostics=[Task failed, taskId=task_1465411930667_0212_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: ORC does not support type conversion from INT to STRING.
I think issue is reflect("java.util.UUID", "randomUUID") AS ID
I know there is Bug raised while reading data from ORC table. Is there any workaround apart from reloading data.
-MS
Re: ORC does not support type conversion from INT to STRING.
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Mathew,
In layman's term if I create the source ORC table column as INT and then
create a target ORC table but that column has now been defined as STRING
and do an INSERT/SELECT from source table how data is internally stored?
Is it implicitly converted into new format using CAST function or it is
stored as is and just masked?
The version of Hive I am using is 2 and it works OK for primitive data
types (insert/select from INT to String)
However, I believe Mahender is referring to Complex types?
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 18 July 2016 at 22:31, Matthew McCline <mm...@hortonworks.com> wrote:
>
> Hi Mahender,
>
>
> Schema Evolution is available on the latest recent version of Hive.
>
>
> For example, if you set
> hive.metastore.disallow.incompatible.col.type.changes=false; on master
> (i.e. hive2) it will support INT to STRING conversion.
>
>
> If you need to remain on an older version, then you are out of luck.
>
>
> Thanks,
>
> Matt
>
>
> ------------------------------
> *From:* Mahender Sarangam <Ma...@outlook.com>
> *Sent:* Monday, July 18, 2016 1:59 PM
> *To:* user@hive.apache.org
> *Subject:* Re: ORC does not support type conversion from INT to STRING.
>
>
> Hi Mich,
>
> Sorry for delay in responding. here is the scenario,
>
> We have created new cluster and we have moved all ORC File data into new
> cluster. We have re-created table pointing to ORC location. We have
> modified data type of ORC table from *INT *to *String.* From then onward,
> we were unable to fire select statement against this ORC table, hive keep
> throwing exception, "Orc table select. Unable to convert Int to String".
> Looks like it is bug in ORC table only. Where in we modify the datatype
> from *int to string,* is causing problem with ORC reading/select
> statement, it throws exceptio. Please let me know if there are any
> workaround for this scenario. Is this behavior expected previously also.
>
>
> */Mahender*
>
>
>
>
>
>
> On 6/14/2016 11:47 AM, Mich Talebzadeh wrote:
>
> you must excuse my ignorance
>
> can you please elaborate on this as there seems something has gone wrong
> somewhere?
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 14 June 2016 at 19:42, Mahender Sarangam <Ma...@outlook.com>
> wrote:
>
>> Yes Mich. We have restored cluster from metastore.
>>
>> On 6/14/2016 11:35 AM, Mich Talebzadeh wrote:
>>
>> Hi Mahendar,
>>
>>
>> Did you load the meta-data DB/schema from backup and now seeing this error
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 14 June 2016 at 19:04, Mahender Sarangam <Mahender.BigData@outlook.com
>> > wrote:
>>
>>> ping.
>>>
>>> On 6/13/2016 1:19 PM, Mahender Sarangam wrote:
>>>
>>> Hi,
>>>
>>> We are facing issue while reading data from ORC table. We have created
>>> ORC table and dumped data into it. We have deleted cluster due to some
>>> reason. When we recreated cluster (using Metastore) and table pointing to
>>> same location. When we perform reading from ORC table. We see below error.
>>>
>>> SELECT col2, Col1,
>>> reflect("java.util.UUID", "randomUUID") AS ID,
>>> Source,
>>> 1 ,
>>> SDate,
>>> EDate
>>> FROM Table ORC JOIN Table2 _surr;
>>>
>>> ERROR : Vertex failed, vertexName=Map 1,
>>> vertexId=vertex_1465411930667_0212_1_01, diagnostics=[Task failed,
>>> taskId=task_1465411930667_0212_1_01_000000, diagnostics=[TaskAttempt 0
>>> failed, info=[Error: Failure while running task:java.lang.RuntimeException:
>>> java.lang.RuntimeException: java.io.IOException: java.io.IOException: ORC
>>> does not support type conversion from INT to STRING.
>>>
>>>
>>> I think issue is reflect("java.util.UUID", "randomUUID") AS ID
>>>
>>>
>>> I know there is Bug raised while reading data from ORC table. Is there
>>> any workaround apart from reloading data.
>>>
>>> -MS
>>>
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: ORC does not support type conversion from INT to STRING.
Posted by Matthew McCline <mm...@hortonworks.com>.
Hi Mahender,
Schema Evolution is available on the latest recent version of Hive.
For example, if you set hive.metastore.disallow.incompatible.col.type.changes=false;? on master (i.e. hive2) it will support INT to STRING conversion.
If you need to remain on an older version, then you are out of luck.
Thanks,
Matt
________________________________
From: Mahender Sarangam <Ma...@outlook.com>
Sent: Monday, July 18, 2016 1:59 PM
To: user@hive.apache.org
Subject: Re: ORC does not support type conversion from INT to STRING.
Hi Mich,
Sorry for delay in responding. here is the scenario,
We have created new cluster and we have moved all ORC File data into new cluster. We have re-created table pointing to ORC location. We have modified data type of ORC table from INT to String. From then onward, we were unable to fire select statement against this ORC table, hive keep throwing exception, "Orc table select. Unable to convert Int to String". Looks like it is bug in ORC table only. Where in we modify the datatype from int to string, is causing problem with ORC reading/select statement, it throws exceptio. Please let me know if there are any workaround for this scenario. Is this behavior expected previously also.
/Mahender
On 6/14/2016 11:47 AM, Mich Talebzadeh wrote:
you must excuse my ignorance
can you please elaborate on this as there seems something has gone wrong somewhere?
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/>
On 14 June 2016 at 19:42, Mahender Sarangam <Ma...@outlook.com>> wrote:
Yes Mich. We have restored cluster from metastore.
On 6/14/2016 11:35 AM, Mich Talebzadeh wrote:
Hi Mahendar,
Did you load the meta-data DB/schema from backup and now seeing this error
Dr Mich Talebzadeh
LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
http://talebzadehmich.wordpress.com
On 14 June 2016 at 19:04, Mahender Sarangam <Ma...@outlook.com>> wrote:
ping.
On 6/13/2016 1:19 PM, Mahender Sarangam wrote:
Hi,
We are facing issue while reading data from ORC table. We have created ORC table and dumped data into it. We have deleted cluster due to some reason. When we recreated cluster (using Metastore) and table pointing to same location. When we perform reading from ORC table. We see below error.
SELECT col2, Col1,
reflect("java.util.UUID", "randomUUID") AS ID,
Source,
1 ,
SDate,
EDate
FROM Table ORC JOIN Table2 _surr;
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1465411930667_0212_1_01, diagnostics=[Task failed, taskId=task_1465411930667_0212_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: ORC does not support type conversion from INT to STRING.
I think issue is reflect("java.util.UUID", "randomUUID") AS ID
I know there is Bug raised while reading data from ORC table. Is there any workaround apart from reloading data.
-MS