You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Walaa Eldin Moustafa <wa...@gmail.com> on 2022/01/27 22:26:36 UTC
Hive table compatibility for Iceberg readers
Hi Iceberg community,
We have been working on converting our tables from the Hive table format to
Iceberg. In order to achieve that switch transparently, we have introduced
a number of Hive table features and compatibility modes in Iceberg, and
connected them to Spark DataSource API. At a high level, given a table name
that exists in both Hive and Iceberg formats (in the same catalog, like
HMS, or in different catalogs behind a meta catalog), this kind of
architecture enables us to change whether the user accesses the Hive or the
Iceberg version of the table just with a property change on the table. We
have used this to incrementally roll out almost all of our Hive tables to
Iceberg readers, in preparation to switching the pointer to read from
physical Iceberg metadata (which we also experimented with). Right now,
those features reside in LinkedIn’s temporary fork of Iceberg (also
open-source [1]). I am wondering if this is something that would be of
interest to the general Iceberg community and if so, we can contribute
those features back. Those features have been in production for a number of
months at LinkedIn. The features include:
1. *Reading tables with only Hive metadata:* If a table was written by
Hive (or Spark DataSource V1), we added a mechanism to construct an Iceberg
in-memory snapshot on the fly from the table directory structure, and let
Iceberg readers read that snapshot.
2. *Iceberg schema lower casing:* Before Iceberg, when users read Hive
tables from Spark, the returned schema is lowercase since Hive stores all
metadata in lowercase mode. If users move to Iceberg, such readers could
break once Iceberg returns proper case schema. This feature is to add
lowercasing for backward compatibility with existing scripts. This feature
is added as an option and is not enabled by default.
3. *Hive table proper casing:* conversely, we leverage the Avro schema
to supplement the lower case Hive schema when reading Hive tables. This is
useful if someone wants to still get proper cased schemas while still in
the Hive mode (to be forward-compatible with Iceberg). The same flag used
in (2) is used here.
4. *Supporting default value semantics:* Hive tables support declaring a
default value in the table schema to use on schema evolution. Currently
Iceberg only supports optional fields, but not fields with default values.
More details on this, and some exceptions that take place are in this
Iceberg issue [2]. The feature is also on the Iceberg project roadmap [3].
5. *Converting union types to structs:* Compute engines have varying
degrees of supporting union types. Some like Spark do not support it on
table APIs. Trino did some work to convert union types to structs [4]. We
have applied the same transformation in Iceberg so those tables can be read
without breaking (in Spark) or restating (of the table). We have connected
the conversion to Spark table APIs, and users can now access tables with
uniontypes from Spark SQL (i.e., both Hive and Iceberg tables). The struct
schema is forward-compatible with Trino on Iceberg, since the schema used
is the same as one used by Trino conversion. This means that current Trino
+ Hive users can transparently leverage that change, and it is a net new
feature in Spark.
What does the community think about those features? It would be great to
get an agreement on features folks think are relevant so we can plan the
work for contributing them back or refactoring them to a separate repo.
[1] https://github.com/linkedin/iceberg
[2] https://github.com/apache/iceberg/issues/2039
[3] https://github.com/apache/iceberg/blob/master/site/docs/roadmap.md
[4] https://github.com/trinodb/trino/pull/3483
Thanks,
Walaa.
Re: Hive table compatibility for Iceberg readers
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
The union type conversion PR is up:
https://github.com/apache/iceberg/pull/4242.
Thanks,
Walaa.
On Fri, Feb 11, 2022 at 8:53 AM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:
> Thanks Ryan! Yes there is an active discussion on the PR on the spec
> aspect.
>
> On Fri, Feb 11, 2022 at 8:47 AM Ryan Blue <bl...@tabular.io> wrote:
>
>> Sounds great. Thanks for the update! That PR is on my list to take a look
>> at, but I still recommend starting with the spec changes. For example, how
>> should default values be stored in Iceberg metadata for each type?
>> Currently, the spec changes just mention defaults without going into detail
>> about how they are tracked and what rules there are about them.
>>
>> On Wed, Feb 9, 2022 at 6:32 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> Thanks Ryan and Owen! Glad we have converged on this. Next steps for us:
>>>
>>> * Continuing the discussion on the default value PR (already ongoing
>>> [1]).
>>> * Filing the union type conversion PR (ETA end of next week).
>>> * Moving listing-based Hive table scan using Iceberg to a separate repo
>>> (likely open source). For this I expect introducing some extension points
>>> to Iceberg such as making some classes SPI. I hope that the community is
>>> okay with that.
>>>
>>> By the way, Owen and I synced on the Hive casing behavior, and it is a
>>> bit more involved: Hive lowers the schema case for all fields (including
>>> nested fields) in the Avro case, but only lowers top-level field case and
>>> preserves inner field case for other formats (we experimented with ORC and
>>> Text). Hope this clarifies the confusion.
>>>
>>> [1] https://github.com/apache/iceberg/pull/2496
>>>
>>> Thanks,
>>> Walaa.
>>>
>>>
>>>
>>> On Wed, Feb 2, 2022 at 2:40 PM Ryan Blue <bl...@tabular.io> wrote:
>>>
>>>> Walaa, thanks for this list. I think most of these are definitely
>>>> useful. I think the best one to focus on first is the default values, since
>>>> those will make Iceberg tables behave more like standard SQL tables, which
>>>> is the goal.
>>>>
>>>> I'm really curious to learn more about #1, but I don't think that I
>>>> have enough detail to know whether it is something that fits in the Iceberg
>>>> project. At Netflix, we had an alternative implementation of Hive and Spark
>>>> tables (Spark tables are slightly different) that we similarly used. But we
>>>> didn't write to both at the same time.
>>>>
>>>> For the others, I'm interested in hearing what other people in the
>>>> community find valuable. I don't think I would use #2 or #3, for example.
>>>> That's because we already support a flag for case insensitive column
>>>> resolution that is well supported throughout Iceberg. If you wanted to use
>>>> alternative names, then I'd probably recommend just turning that on...
>>>> although that may not be an option depending on how you're working with a
>>>> table. It would work in Spark, though. This may be a better feature for
>>>> your system that is built on Iceberg.
>>>>
>>>> Reading unions as structs has come up a couple times so that seems like
>>>> people will want it. I think someone attempted to add this support in the
>>>> past, but ran into issues because the spec is clear that these are NOT
>>>> Iceberg files. There is no guarantee that other implementations will read
>>>> them and Iceberg cannot write them in this form. I'm fairly confident that
>>>> not allowing unions to be written is a good choice, but I would support
>>>> being able to read them.
>>>>
>>>> Ryan
>>>>
>>>> On Mon, Jan 31, 2022 at 4:32 PM Owen O'Malley <ow...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Jan 27, 2022 at 10:26 PM Walaa Eldin Moustafa <
>>>>> wa.moustafa@gmail.com> wrote:
>>>>>
>>>>>> *2. Iceberg schema lower casing:* Before Iceberg, when users read
>>>>>> Hive tables from Spark, the returned schema is lowercase since Hive stores
>>>>>> all metadata in lowercase mode. If users move to Iceberg, such readers
>>>>>> could break once Iceberg returns proper case schema. This feature is to add
>>>>>> lowercasing for backward compatibility with existing scripts. This feature
>>>>>> is added as an option and is not enabled by default.
>>>>>>
>>>>>
>>>>> This isn't quite correct. Hive lowercases top-level columns. It does
>>>>> not lowercase field names inside structs.
>>>>>
>>>>>
>>>>>> *3. Hive table proper casing:* conversely, we leverage the Avro
>>>>>> schema to supplement the lower case Hive schema when reading Hive tables.
>>>>>> This is useful if someone wants to still get proper cased schemas while
>>>>>> still in the Hive mode (to be forward-compatible with Iceberg). The same
>>>>>> flag used in (2) is used here.
>>>>>>
>>>>>
>>>>> Are there users of Avro schemas in Hive outside of LinkedIn? I've
>>>>> never seen it used. I don't think you should tie #2 and #3 together.
>>>>>
>>>>> Supporting default values and union types are useful extensions.
>>>>>
>>>>> .. Owen
>>>>>
>>>>
>>>>
>>>> --
>>>> Ryan Blue
>>>> Tabular
>>>>
>>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>
Re: Hive table compatibility for Iceberg readers
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
Thanks Ryan! Yes there is an active discussion on the PR on the spec aspect.
On Fri, Feb 11, 2022 at 8:47 AM Ryan Blue <bl...@tabular.io> wrote:
> Sounds great. Thanks for the update! That PR is on my list to take a look
> at, but I still recommend starting with the spec changes. For example, how
> should default values be stored in Iceberg metadata for each type?
> Currently, the spec changes just mention defaults without going into detail
> about how they are tracked and what rules there are about them.
>
> On Wed, Feb 9, 2022 at 6:32 PM Walaa Eldin Moustafa <wa...@gmail.com>
> wrote:
>
>> Thanks Ryan and Owen! Glad we have converged on this. Next steps for us:
>>
>> * Continuing the discussion on the default value PR (already ongoing [1]).
>> * Filing the union type conversion PR (ETA end of next week).
>> * Moving listing-based Hive table scan using Iceberg to a separate repo
>> (likely open source). For this I expect introducing some extension points
>> to Iceberg such as making some classes SPI. I hope that the community is
>> okay with that.
>>
>> By the way, Owen and I synced on the Hive casing behavior, and it is a
>> bit more involved: Hive lowers the schema case for all fields (including
>> nested fields) in the Avro case, but only lowers top-level field case and
>> preserves inner field case for other formats (we experimented with ORC and
>> Text). Hope this clarifies the confusion.
>>
>> [1] https://github.com/apache/iceberg/pull/2496
>>
>> Thanks,
>> Walaa.
>>
>>
>>
>> On Wed, Feb 2, 2022 at 2:40 PM Ryan Blue <bl...@tabular.io> wrote:
>>
>>> Walaa, thanks for this list. I think most of these are definitely
>>> useful. I think the best one to focus on first is the default values, since
>>> those will make Iceberg tables behave more like standard SQL tables, which
>>> is the goal.
>>>
>>> I'm really curious to learn more about #1, but I don't think that I have
>>> enough detail to know whether it is something that fits in the Iceberg
>>> project. At Netflix, we had an alternative implementation of Hive and Spark
>>> tables (Spark tables are slightly different) that we similarly used. But we
>>> didn't write to both at the same time.
>>>
>>> For the others, I'm interested in hearing what other people in the
>>> community find valuable. I don't think I would use #2 or #3, for example.
>>> That's because we already support a flag for case insensitive column
>>> resolution that is well supported throughout Iceberg. If you wanted to use
>>> alternative names, then I'd probably recommend just turning that on...
>>> although that may not be an option depending on how you're working with a
>>> table. It would work in Spark, though. This may be a better feature for
>>> your system that is built on Iceberg.
>>>
>>> Reading unions as structs has come up a couple times so that seems like
>>> people will want it. I think someone attempted to add this support in the
>>> past, but ran into issues because the spec is clear that these are NOT
>>> Iceberg files. There is no guarantee that other implementations will read
>>> them and Iceberg cannot write them in this form. I'm fairly confident that
>>> not allowing unions to be written is a good choice, but I would support
>>> being able to read them.
>>>
>>> Ryan
>>>
>>> On Mon, Jan 31, 2022 at 4:32 PM Owen O'Malley <ow...@gmail.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Jan 27, 2022 at 10:26 PM Walaa Eldin Moustafa <
>>>> wa.moustafa@gmail.com> wrote:
>>>>
>>>>> *2. Iceberg schema lower casing:* Before Iceberg, when users read
>>>>> Hive tables from Spark, the returned schema is lowercase since Hive stores
>>>>> all metadata in lowercase mode. If users move to Iceberg, such readers
>>>>> could break once Iceberg returns proper case schema. This feature is to add
>>>>> lowercasing for backward compatibility with existing scripts. This feature
>>>>> is added as an option and is not enabled by default.
>>>>>
>>>>
>>>> This isn't quite correct. Hive lowercases top-level columns. It does
>>>> not lowercase field names inside structs.
>>>>
>>>>
>>>>> *3. Hive table proper casing:* conversely, we leverage the Avro
>>>>> schema to supplement the lower case Hive schema when reading Hive tables.
>>>>> This is useful if someone wants to still get proper cased schemas while
>>>>> still in the Hive mode (to be forward-compatible with Iceberg). The same
>>>>> flag used in (2) is used here.
>>>>>
>>>>
>>>> Are there users of Avro schemas in Hive outside of LinkedIn? I've never
>>>> seen it used. I don't think you should tie #2 and #3 together.
>>>>
>>>> Supporting default values and union types are useful extensions.
>>>>
>>>> .. Owen
>>>>
>>>
>>>
>>> --
>>> Ryan Blue
>>> Tabular
>>>
>>
>
> --
> Ryan Blue
> Tabular
>
Re: Hive table compatibility for Iceberg readers
Posted by Ryan Blue <bl...@tabular.io>.
Sounds great. Thanks for the update! That PR is on my list to take a look
at, but I still recommend starting with the spec changes. For example, how
should default values be stored in Iceberg metadata for each type?
Currently, the spec changes just mention defaults without going into detail
about how they are tracked and what rules there are about them.
On Wed, Feb 9, 2022 at 6:32 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:
> Thanks Ryan and Owen! Glad we have converged on this. Next steps for us:
>
> * Continuing the discussion on the default value PR (already ongoing [1]).
> * Filing the union type conversion PR (ETA end of next week).
> * Moving listing-based Hive table scan using Iceberg to a separate repo
> (likely open source). For this I expect introducing some extension points
> to Iceberg such as making some classes SPI. I hope that the community is
> okay with that.
>
> By the way, Owen and I synced on the Hive casing behavior, and it is a bit
> more involved: Hive lowers the schema case for all fields (including nested
> fields) in the Avro case, but only lowers top-level field case and
> preserves inner field case for other formats (we experimented with ORC and
> Text). Hope this clarifies the confusion.
>
> [1] https://github.com/apache/iceberg/pull/2496
>
> Thanks,
> Walaa.
>
>
>
> On Wed, Feb 2, 2022 at 2:40 PM Ryan Blue <bl...@tabular.io> wrote:
>
>> Walaa, thanks for this list. I think most of these are definitely useful.
>> I think the best one to focus on first is the default values, since those
>> will make Iceberg tables behave more like standard SQL tables, which is the
>> goal.
>>
>> I'm really curious to learn more about #1, but I don't think that I have
>> enough detail to know whether it is something that fits in the Iceberg
>> project. At Netflix, we had an alternative implementation of Hive and Spark
>> tables (Spark tables are slightly different) that we similarly used. But we
>> didn't write to both at the same time.
>>
>> For the others, I'm interested in hearing what other people in the
>> community find valuable. I don't think I would use #2 or #3, for example.
>> That's because we already support a flag for case insensitive column
>> resolution that is well supported throughout Iceberg. If you wanted to use
>> alternative names, then I'd probably recommend just turning that on...
>> although that may not be an option depending on how you're working with a
>> table. It would work in Spark, though. This may be a better feature for
>> your system that is built on Iceberg.
>>
>> Reading unions as structs has come up a couple times so that seems like
>> people will want it. I think someone attempted to add this support in the
>> past, but ran into issues because the spec is clear that these are NOT
>> Iceberg files. There is no guarantee that other implementations will read
>> them and Iceberg cannot write them in this form. I'm fairly confident that
>> not allowing unions to be written is a good choice, but I would support
>> being able to read them.
>>
>> Ryan
>>
>> On Mon, Jan 31, 2022 at 4:32 PM Owen O'Malley <ow...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Jan 27, 2022 at 10:26 PM Walaa Eldin Moustafa <
>>> wa.moustafa@gmail.com> wrote:
>>>
>>>> *2. Iceberg schema lower casing:* Before Iceberg, when users read Hive
>>>> tables from Spark, the returned schema is lowercase since Hive stores all
>>>> metadata in lowercase mode. If users move to Iceberg, such readers could
>>>> break once Iceberg returns proper case schema. This feature is to add
>>>> lowercasing for backward compatibility with existing scripts. This feature
>>>> is added as an option and is not enabled by default.
>>>>
>>>
>>> This isn't quite correct. Hive lowercases top-level columns. It does not
>>> lowercase field names inside structs.
>>>
>>>
>>>> *3. Hive table proper casing:* conversely, we leverage the Avro schema
>>>> to supplement the lower case Hive schema when reading Hive tables. This is
>>>> useful if someone wants to still get proper cased schemas while still in
>>>> the Hive mode (to be forward-compatible with Iceberg). The same flag used
>>>> in (2) is used here.
>>>>
>>>
>>> Are there users of Avro schemas in Hive outside of LinkedIn? I've never
>>> seen it used. I don't think you should tie #2 and #3 together.
>>>
>>> Supporting default values and union types are useful extensions.
>>>
>>> .. Owen
>>>
>>
>>
>> --
>> Ryan Blue
>> Tabular
>>
>
--
Ryan Blue
Tabular
Re: Hive table compatibility for Iceberg readers
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
Thanks Ryan and Owen! Glad we have converged on this. Next steps for us:
* Continuing the discussion on the default value PR (already ongoing [1]).
* Filing the union type conversion PR (ETA end of next week).
* Moving listing-based Hive table scan using Iceberg to a separate repo
(likely open source). For this I expect introducing some extension points
to Iceberg such as making some classes SPI. I hope that the community is
okay with that.
By the way, Owen and I synced on the Hive casing behavior, and it is a bit
more involved: Hive lowers the schema case for all fields (including nested
fields) in the Avro case, but only lowers top-level field case and
preserves inner field case for other formats (we experimented with ORC and
Text). Hope this clarifies the confusion.
[1] https://github.com/apache/iceberg/pull/2496
Thanks,
Walaa.
On Wed, Feb 2, 2022 at 2:40 PM Ryan Blue <bl...@tabular.io> wrote:
> Walaa, thanks for this list. I think most of these are definitely useful.
> I think the best one to focus on first is the default values, since those
> will make Iceberg tables behave more like standard SQL tables, which is the
> goal.
>
> I'm really curious to learn more about #1, but I don't think that I have
> enough detail to know whether it is something that fits in the Iceberg
> project. At Netflix, we had an alternative implementation of Hive and Spark
> tables (Spark tables are slightly different) that we similarly used. But we
> didn't write to both at the same time.
>
> For the others, I'm interested in hearing what other people in the
> community find valuable. I don't think I would use #2 or #3, for example.
> That's because we already support a flag for case insensitive column
> resolution that is well supported throughout Iceberg. If you wanted to use
> alternative names, then I'd probably recommend just turning that on...
> although that may not be an option depending on how you're working with a
> table. It would work in Spark, though. This may be a better feature for
> your system that is built on Iceberg.
>
> Reading unions as structs has come up a couple times so that seems like
> people will want it. I think someone attempted to add this support in the
> past, but ran into issues because the spec is clear that these are NOT
> Iceberg files. There is no guarantee that other implementations will read
> them and Iceberg cannot write them in this form. I'm fairly confident that
> not allowing unions to be written is a good choice, but I would support
> being able to read them.
>
> Ryan
>
> On Mon, Jan 31, 2022 at 4:32 PM Owen O'Malley <ow...@gmail.com>
> wrote:
>
>>
>>
>> On Thu, Jan 27, 2022 at 10:26 PM Walaa Eldin Moustafa <
>> wa.moustafa@gmail.com> wrote:
>>
>>> *2. Iceberg schema lower casing:* Before Iceberg, when users read Hive
>>> tables from Spark, the returned schema is lowercase since Hive stores all
>>> metadata in lowercase mode. If users move to Iceberg, such readers could
>>> break once Iceberg returns proper case schema. This feature is to add
>>> lowercasing for backward compatibility with existing scripts. This feature
>>> is added as an option and is not enabled by default.
>>>
>>
>> This isn't quite correct. Hive lowercases top-level columns. It does not
>> lowercase field names inside structs.
>>
>>
>>> *3. Hive table proper casing:* conversely, we leverage the Avro schema
>>> to supplement the lower case Hive schema when reading Hive tables. This is
>>> useful if someone wants to still get proper cased schemas while still in
>>> the Hive mode (to be forward-compatible with Iceberg). The same flag used
>>> in (2) is used here.
>>>
>>
>> Are there users of Avro schemas in Hive outside of LinkedIn? I've never
>> seen it used. I don't think you should tie #2 and #3 together.
>>
>> Supporting default values and union types are useful extensions.
>>
>> .. Owen
>>
>
>
> --
> Ryan Blue
> Tabular
>
Re: Hive table compatibility for Iceberg readers
Posted by Ryan Blue <bl...@tabular.io>.
Walaa, thanks for this list. I think most of these are definitely useful. I
think the best one to focus on first is the default values, since those
will make Iceberg tables behave more like standard SQL tables, which is the
goal.
I'm really curious to learn more about #1, but I don't think that I have
enough detail to know whether it is something that fits in the Iceberg
project. At Netflix, we had an alternative implementation of Hive and Spark
tables (Spark tables are slightly different) that we similarly used. But we
didn't write to both at the same time.
For the others, I'm interested in hearing what other people in the
community find valuable. I don't think I would use #2 or #3, for example.
That's because we already support a flag for case insensitive column
resolution that is well supported throughout Iceberg. If you wanted to use
alternative names, then I'd probably recommend just turning that on...
although that may not be an option depending on how you're working with a
table. It would work in Spark, though. This may be a better feature for
your system that is built on Iceberg.
Reading unions as structs has come up a couple times so that seems like
people will want it. I think someone attempted to add this support in the
past, but ran into issues because the spec is clear that these are NOT
Iceberg files. There is no guarantee that other implementations will read
them and Iceberg cannot write them in this form. I'm fairly confident that
not allowing unions to be written is a good choice, but I would support
being able to read them.
Ryan
On Mon, Jan 31, 2022 at 4:32 PM Owen O'Malley <ow...@gmail.com>
wrote:
>
>
> On Thu, Jan 27, 2022 at 10:26 PM Walaa Eldin Moustafa <
> wa.moustafa@gmail.com> wrote:
>
>> *2. Iceberg schema lower casing:* Before Iceberg, when users read Hive
>> tables from Spark, the returned schema is lowercase since Hive stores all
>> metadata in lowercase mode. If users move to Iceberg, such readers could
>> break once Iceberg returns proper case schema. This feature is to add
>> lowercasing for backward compatibility with existing scripts. This feature
>> is added as an option and is not enabled by default.
>>
>
> This isn't quite correct. Hive lowercases top-level columns. It does not
> lowercase field names inside structs.
>
>
>> *3. Hive table proper casing:* conversely, we leverage the Avro schema
>> to supplement the lower case Hive schema when reading Hive tables. This is
>> useful if someone wants to still get proper cased schemas while still in
>> the Hive mode (to be forward-compatible with Iceberg). The same flag used
>> in (2) is used here.
>>
>
> Are there users of Avro schemas in Hive outside of LinkedIn? I've never
> seen it used. I don't think you should tie #2 and #3 together.
>
> Supporting default values and union types are useful extensions.
>
> .. Owen
>
--
Ryan Blue
Tabular
Re: Hive table compatibility for Iceberg readers
Posted by Owen O'Malley <ow...@gmail.com>.
On Thu, Jan 27, 2022 at 10:26 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:
> *2. Iceberg schema lower casing:* Before Iceberg, when users read Hive
> tables from Spark, the returned schema is lowercase since Hive stores all
> metadata in lowercase mode. If users move to Iceberg, such readers could
> break once Iceberg returns proper case schema. This feature is to add
> lowercasing for backward compatibility with existing scripts. This feature
> is added as an option and is not enabled by default.
>
This isn't quite correct. Hive lowercases top-level columns. It does not
lowercase field names inside structs.
> *3. Hive table proper casing:* conversely, we leverage the Avro schema to
> supplement the lower case Hive schema when reading Hive tables. This is
> useful if someone wants to still get proper cased schemas while still in
> the Hive mode (to be forward-compatible with Iceberg). The same flag used
> in (2) is used here.
>
Are there users of Avro schemas in Hive outside of LinkedIn? I've never
seen it used. I don't think you should tie #2 and #3 together.
Supporting default values and union types are useful extensions.
.. Owen
Re: Hive table compatibility for Iceberg readers
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
Hi everyone, bumping up this thread. Does it help to discuss each feature
individually?
I am guessing (4. Default value support) is straightforward since it is on
the roadmap already. If not, let us start with this.
As for the next step, it would be great if we can focus on (1. Reading
tables with only Hive metadata) and (5. Supporting files that contain
uniontypes). It is worth noting that (5) does not require Hive table
support, i.e., (1), since it can occur if the files with uniontype values
are reused in Iceberg manifests. On the other hand, we have already started
moving (1) to another repo, but we could save some steps if folks here
think it is worth moving to Iceberg.
Thanks,
Walaa.
On Thu, Jan 27, 2022 at 2:26 PM Walaa Eldin Moustafa <wa...@gmail.com>
wrote:
> Hi Iceberg community,
>
> We have been working on converting our tables from the Hive table format
> to Iceberg. In order to achieve that switch transparently, we have
> introduced a number of Hive table features and compatibility modes in
> Iceberg, and connected them to Spark DataSource API. At a high level, given
> a table name that exists in both Hive and Iceberg formats (in the same
> catalog, like HMS, or in different catalogs behind a meta catalog), this
> kind of architecture enables us to change whether the user accesses the
> Hive or the Iceberg version of the table just with a property change on the
> table. We have used this to incrementally roll out almost all of our Hive
> tables to Iceberg readers, in preparation to switching the pointer to read
> from physical Iceberg metadata (which we also experimented with). Right
> now, those features reside in LinkedIn’s temporary fork of Iceberg (also
> open-source [1]). I am wondering if this is something that would be of
> interest to the general Iceberg community and if so, we can contribute
> those features back. Those features have been in production for a number of
> months at LinkedIn. The features include:
>
> 1. *Reading tables with only Hive metadata:* If a table was written by
> Hive (or Spark DataSource V1), we added a mechanism to construct an Iceberg
> in-memory snapshot on the fly from the table directory structure, and let
> Iceberg readers read that snapshot.
> 2. *Iceberg schema lower casing:* Before Iceberg, when users read Hive
> tables from Spark, the returned schema is lowercase since Hive stores all
> metadata in lowercase mode. If users move to Iceberg, such readers could
> break once Iceberg returns proper case schema. This feature is to add
> lowercasing for backward compatibility with existing scripts. This feature
> is added as an option and is not enabled by default.
> 3. *Hive table proper casing:* conversely, we leverage the Avro schema
> to supplement the lower case Hive schema when reading Hive tables. This is
> useful if someone wants to still get proper cased schemas while still in
> the Hive mode (to be forward-compatible with Iceberg). The same flag used
> in (2) is used here.
> 4. *Supporting default value semantics:* Hive tables support declaring
> a default value in the table schema to use on schema evolution. Currently
> Iceberg only supports optional fields, but not fields with default values.
> More details on this, and some exceptions that take place are in this
> Iceberg issue [2]. The feature is also on the Iceberg project roadmap [3].
> 5. *Converting union types to structs:* Compute engines have varying
> degrees of supporting union types. Some like Spark do not support it on
> table APIs. Trino did some work to convert union types to structs [4]. We
> have applied the same transformation in Iceberg so those tables can be read
> without breaking (in Spark) or restating (of the table). We have connected
> the conversion to Spark table APIs, and users can now access tables with
> uniontypes from Spark SQL (i.e., both Hive and Iceberg tables). The struct
> schema is forward-compatible with Trino on Iceberg, since the schema used
> is the same as one used by Trino conversion. This means that current Trino
> + Hive users can transparently leverage that change, and it is a net new
> feature in Spark.
>
> What does the community think about those features? It would be great to
> get an agreement on features folks think are relevant so we can plan the
> work for contributing them back or refactoring them to a separate repo.
>
> [1] https://github.com/linkedin/iceberg
> [2] https://github.com/apache/iceberg/issues/2039
> [3] https://github.com/apache/iceberg/blob/master/site/docs/roadmap.md
> [4] https://github.com/trinodb/trino/pull/3483
>
> Thanks,
> Walaa.
>
>