You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@orc.apache.org by Owen O'Malley <ow...@gmail.com> on 2019/03/01 23:19:46 UTC
Re: WriterOptions.writerVersion(version)?
The goal of WriterVersion is to record changes to the writer software so
that the readers can cope with unknown bugs. It is not intended to mark
format changes. A good example of this is when we switched from the
row-by-row writer to the vectorized writer in HIVE-12055. This changed the
implementation of the writer, but didn't change the format. If the change
had introduced a bug, we'd know that the reader had to compensate.
If the older versions of Hive are broken with higher writer versions, we
absolutely should fix that. I seem to remember fixing that at some point,
but I probably didn't push it back into Hive 1.x. Which version did you see
the problem?
.. Owen
On Wed, Feb 27, 2019 at 9:43 AM Dain Sundstrom <da...@iq80.com> wrote:
> Hi, we recently updated to Hive 3.0+ and have noticed some issues with
> older versions of Hive being able to read data written by newer versions of
> ORC. Specifically, older readers only understand writer version up to 4
> and newer versions write 6. This causes older readers to fail. I see that
> the workaround is to set
> `WriterOptions.writerVersion(WriterVersion.HIVE_13083)`, which causes the
> writer to put a `4` in the postscript, but doesn’t seem to change anything
> else in the writer’s behavior. My question is, did I miss something gin
> the writer where behavior changes based on version? If not, does that
> work? I ask because newer versions have comments like `ORC_135(6) =>
> timestamp stats use utc`, which to me would seem to require that the
> behavior changes.
>
> Thanks,
>
> -dain
>
> ----
> Dain Sundstrom
> Co-founder @ Presto Software Foundation, Co-creator of Presto (
> https://prestosql.io)
>
>
Re: WriterOptions.writerVersion(version)?
Posted by Dain Sundstrom <da...@iq80.com>.
After a bit more investigation, it looks like this was a regression only present in Hive 2.0 to 2.2.
----
Dain Sundstrom
Co-founder @ Presto Software Foundation, Co-creator of Presto (https://prestosql.io)
> On Mar 1, 2019, at 5:58 PM, Dain Sundstrom <da...@iq80.com> wrote:
>
> Thanks Owen,
>
> I’m not the one working on this directly, but the folks looking at this said:
>
>> This was fixed in ORC-125 which made it into Hive 2.3 via HIVE-14007. I believe this affects all earlier Hive 2.x versions but I didn't confirm that.
>
> So it looks like you fixed it 2 years ago.
>
> ----
> Dain Sundstrom
> Co-founder @ Presto Software Foundation, Co-creator of Presto (https://prestosql.io)
>
>> On Mar 1, 2019, at 3:19 PM, Owen O'Malley <ow...@gmail.com> wrote:
>>
>> The goal of WriterVersion is to record changes to the writer software so
>> that the readers can cope with unknown bugs. It is not intended to mark
>> format changes. A good example of this is when we switched from the
>> row-by-row writer to the vectorized writer in HIVE-12055. This changed the
>> implementation of the writer, but didn't change the format. If the change
>> had introduced a bug, we'd know that the reader had to compensate.
>>
>> If the older versions of Hive are broken with higher writer versions, we
>> absolutely should fix that. I seem to remember fixing that at some point,
>> but I probably didn't push it back into Hive 1.x. Which version did you see
>> the problem?
>>
>> .. Owen
>>
>> On Wed, Feb 27, 2019 at 9:43 AM Dain Sundstrom <da...@iq80.com> wrote:
>>
>>> Hi, we recently updated to Hive 3.0+ and have noticed some issues with
>>> older versions of Hive being able to read data written by newer versions of
>>> ORC. Specifically, older readers only understand writer version up to 4
>>> and newer versions write 6. This causes older readers to fail. I see that
>>> the workaround is to set
>>> `WriterOptions.writerVersion(WriterVersion.HIVE_13083)`, which causes the
>>> writer to put a `4` in the postscript, but doesn’t seem to change anything
>>> else in the writer’s behavior. My question is, did I miss something gin
>>> the writer where behavior changes based on version? If not, does that
>>> work? I ask because newer versions have comments like `ORC_135(6) =>
>>> timestamp stats use utc`, which to me would seem to require that the
>>> behavior changes.
>>>
>>> Thanks,
>>>
>>> -dain
>>>
>>> ----
>>> Dain Sundstrom
>>> Co-founder @ Presto Software Foundation, Co-creator of Presto (
>>> https://prestosql.io)
>>>
>>>
>
Re: WriterOptions.writerVersion(version)?
Posted by Dain Sundstrom <da...@iq80.com>.
Thanks Owen,
I’m not the one working on this directly, but the folks looking at this said:
> This was fixed in ORC-125 which made it into Hive 2.3 via HIVE-14007. I believe this affects all earlier Hive 2.x versions but I didn't confirm that.
So it looks like you fixed it 2 years ago.
----
Dain Sundstrom
Co-founder @ Presto Software Foundation, Co-creator of Presto (https://prestosql.io)
> On Mar 1, 2019, at 3:19 PM, Owen O'Malley <ow...@gmail.com> wrote:
>
> The goal of WriterVersion is to record changes to the writer software so
> that the readers can cope with unknown bugs. It is not intended to mark
> format changes. A good example of this is when we switched from the
> row-by-row writer to the vectorized writer in HIVE-12055. This changed the
> implementation of the writer, but didn't change the format. If the change
> had introduced a bug, we'd know that the reader had to compensate.
>
> If the older versions of Hive are broken with higher writer versions, we
> absolutely should fix that. I seem to remember fixing that at some point,
> but I probably didn't push it back into Hive 1.x. Which version did you see
> the problem?
>
> .. Owen
>
> On Wed, Feb 27, 2019 at 9:43 AM Dain Sundstrom <da...@iq80.com> wrote:
>
>> Hi, we recently updated to Hive 3.0+ and have noticed some issues with
>> older versions of Hive being able to read data written by newer versions of
>> ORC. Specifically, older readers only understand writer version up to 4
>> and newer versions write 6. This causes older readers to fail. I see that
>> the workaround is to set
>> `WriterOptions.writerVersion(WriterVersion.HIVE_13083)`, which causes the
>> writer to put a `4` in the postscript, but doesn’t seem to change anything
>> else in the writer’s behavior. My question is, did I miss something gin
>> the writer where behavior changes based on version? If not, does that
>> work? I ask because newer versions have comments like `ORC_135(6) =>
>> timestamp stats use utc`, which to me would seem to require that the
>> behavior changes.
>>
>> Thanks,
>>
>> -dain
>>
>> ----
>> Dain Sundstrom
>> Co-founder @ Presto Software Foundation, Co-creator of Presto (
>> https://prestosql.io)
>>
>>