You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by William Shen <wi...@marinsoftware.com> on 2019/07/17 23:14:30 UTC
Converting Hive Column from Varchar to String
Hi all,
I assumed that it should be compatible to convert column type varchar to
string, however, after running ALTER TABLE table CHANGE col col STRING, I
encounter the following error when querying the column from hive:
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.io.Text
Anyone encountered this before, or know how to work around this?
Thank you!
- Will
Re: Converting Hive Column from Varchar to String
Posted by William Shen <wi...@marinsoftware.com>.
To add, the storage information is as follows
# Storage Information
SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
Compressed: No
On Wed, Jul 17, 2019 at 4:14 PM William Shen <wi...@marinsoftware.com>
wrote:
> Hi all,
>
> I assumed that it should be compatible to convert column type varchar to
> string, however, after running ALTER TABLE table CHANGE col col STRING, I
> encounter the following error when querying the column from hive:
>
> Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.io.Text
>
> Anyone encountered this before, or know how to work around this?
>
> Thank you!
>
> - Will
>
Re: Converting Hive Column from Varchar to String
Posted by Owen O'Malley <ow...@gmail.com>.
Which version of Hive are you on? The recent versions (hive >= 2.3) should
support schema evolution in the ORC reader.
.. Owen
On Wed, Jul 17, 2019 at 11:07 PM Jörn Franke <jo...@gmail.com> wrote:
> You have to create a new table with this column as varchar and do a select
> insert from the old table.
>
> > Am 18.07.2019 um 01:14 schrieb William Shen <willshen@marinsoftware.com
> >:
> >
> > Hi all,
> >
> > I assumed that it should be compatible to convert column type varchar to
> string, however, after running ALTER TABLE table CHANGE col col STRING, I
> encounter the following error when querying the column from hive:
> >
> > Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.io.Text
> >
> > Anyone encountered this before, or know how to work around this?
> >
> > Thank you!
> >
> > - Will
>
Re: Converting Hive Column from Varchar to String
Posted by Jörn Franke <jo...@gmail.com>.
You have to create a new table with this column as varchar and do a select insert from the old table.
> Am 18.07.2019 um 01:14 schrieb William Shen <wi...@marinsoftware.com>:
>
> Hi all,
>
> I assumed that it should be compatible to convert column type varchar to string, however, after running ALTER TABLE table CHANGE col col STRING, I encounter the following error when querying the column from hive:
>
> Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.io.Text
>
> Anyone encountered this before, or know how to work around this?
>
> Thank you!
>
> - Will
Re: Converting Hive Column from Varchar to String
Posted by William Shen <wi...@marinsoftware.com>.
Thank you everyone for your help! Owen, we're on an old version of hive
(1.1.0-cdh5.9.2).
On Thu, Jul 18, 2019 at 9:38 AM Owen O'Malley <ow...@gmail.com>
wrote:
> ORC files expect UTF-8, which is a superset of ascii, in strings, char,
> and varchar. The only place that I know that will cause trouble if you put
> non-utf-8 data in strings is the statistics. The API for getting the
> min/max will convert to Java strings.
>
> But back to your original point, the schema evolution should easily handle
> the case of string to varchar. Which version of Hive are you using?
>
> .. Owen
>
> On Thu, Jul 18, 2019 at 8:27 AM Devopam Mittra <de...@gmail.com> wrote:
>
>> The table has data in it perhaps that is beyond ASCII.
>> Easier way is to go for additional column , update with data and the drop
>> the older one after validation of records in String type col.
>>
>> Regards
>> Dev
>>
>> On Thu, Jul 18, 2019, 4:44 AM William Shen <wi...@marinsoftware.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I assumed that it should be compatible to convert column type varchar to
>>> string, however, after running ALTER TABLE table CHANGE col col STRING, I
>>> encounter the following error when querying the column from hive:
>>>
>>> Failed with exception
>>> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
>>> java.lang.ClassCastException:
>>> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
>>> org.apache.hadoop.io.Text
>>>
>>> Anyone encountered this before, or know how to work around this?
>>>
>>> Thank you!
>>>
>>> - Will
>>>
>>
Re: Converting Hive Column from Varchar to String
Posted by Owen O'Malley <ow...@gmail.com>.
ORC files expect UTF-8, which is a superset of ascii, in strings, char, and
varchar. The only place that I know that will cause trouble if you put
non-utf-8 data in strings is the statistics. The API for getting the
min/max will convert to Java strings.
But back to your original point, the schema evolution should easily handle
the case of string to varchar. Which version of Hive are you using?
.. Owen
On Thu, Jul 18, 2019 at 8:27 AM Devopam Mittra <de...@gmail.com> wrote:
> The table has data in it perhaps that is beyond ASCII.
> Easier way is to go for additional column , update with data and the drop
> the older one after validation of records in String type col.
>
> Regards
> Dev
>
> On Thu, Jul 18, 2019, 4:44 AM William Shen <wi...@marinsoftware.com>
> wrote:
>
>> Hi all,
>>
>> I assumed that it should be compatible to convert column type varchar to
>> string, however, after running ALTER TABLE table CHANGE col col STRING, I
>> encounter the following error when querying the column from hive:
>>
>> Failed with exception
>> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
>> java.lang.ClassCastException:
>> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
>> org.apache.hadoop.io.Text
>>
>> Anyone encountered this before, or know how to work around this?
>>
>> Thank you!
>>
>> - Will
>>
>
Re: Converting Hive Column from Varchar to String
Posted by Devopam Mittra <de...@gmail.com>.
The table has data in it perhaps that is beyond ASCII.
Easier way is to go for additional column , update with data and the drop
the older one after validation of records in String type col.
Regards
Dev
On Thu, Jul 18, 2019, 4:44 AM William Shen <wi...@marinsoftware.com>
wrote:
> Hi all,
>
> I assumed that it should be compatible to convert column type varchar to
> string, however, after running ALTER TABLE table CHANGE col col STRING, I
> encounter the following error when querying the column from hive:
>
> Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.io.Text
>
> Anyone encountered this before, or know how to work around this?
>
> Thank you!
>
> - Will
>