Re: [DISCUSS] Implementation strategies for supporting Iceberg tables in Hive

[Owen O'Malley <ow...@gmail.com>]

> On Jul 24, 2019, at 22:52, Adrien Guillo <ad...@airbnb.com.invalid> wrote:
> 
> Hi Iceberg folks,
> 
> In the last few months, we (the data infrastructure team at Airbnb) have been closely following the project. We are currently evaluating potential strategies to migrate our data warehouse to Iceberg. However, we have a very large Hive deployment, which means we can’t really do so without support for Iceberg tables in Hive.
> 
> We have been thinking about implementation strategies. Here are some thoughts that we would like to share them with you:
> 
> – Implementing a new `RawStore`

My thought would be to use the embedded metastore with an iceberg rawstore. That enables the client to do the work rather than pushing it to an external metastore. 

I expect that some new users would be able to just use iceberg as their only metastore, but that others would want to have some databases in hive layout and others in iceberg. We could use a delegating raw store for that.

> This is something that has been mentioned several times on the mailing list and seems to indicate that adding support for Iceberg tables in Hive could be achieved without client-side modifications. Does that mean that the Metastore is the only process manipulating Iceberg metadata (snapshots, manifests)? Does that mean that for instance the `listPartition*` calls to the Metastore return the DataFiles associated with each partition? Per our understanding, it seems that supporting Iceberg tables in Hive with this strategy will most likely require to update the RawStore interface AND will require at least some client-side changes. In addition, with this strategy the Metastore bears new responsibilities, which contradicts one of the Iceberg design goals: offloading more work to jobs and removing the metastore as a bottleneck. In the Iceberg world, not much is needed from the Metastore: it just keeps track of the metadata location and provides a mechanism for atomically updating this location (basically, what is done in the `HiveTableOperations` class). We would like to design a solution that relies  as little as possible on the Metastore so that in future we have the option to replace our fleet of Metastores with a simpler system.
> 
> 
> – Implementing a new `HiveStorageHandler`

With an iceberg raw store, I suspect that you might not need a storage handler and could go straight to a input/output format. You would probably need an input and output format for each of the storage formats: Iceberg{Orc,Parquet,Avro}{Input,Output}Format.

.. Owen
> 
> We are working on implementing custom `InputFormat` and `OutputFormat` classes for Iceberg (more on that in the next paragraph) and they would fit in nicely with the `HiveStorageHandler` and `HiveStoragePredicateHandler` interfaces. However, the `HiveMetaHook` interface does not seem rich enough to accommodate all the workflows, for instance no hooks run on `ALTER ...`  or `INSERT...` commands.
> 
> 
> 
> – Proof of concept
> 
> We set out to write a proof of concept that would allow us to learn and experiment. We based our work on the 2.3 branch. Here’s the state of the project and the paths we explored:
> 
> DDL commands
> We support some commands such as `CREABLE TABLE ...`, `DESC ...`, `SHOW PARTITIONS`. They are all implemented in the client and mostly rely on the `HiveCatalog` class to do the work.
> 
> Read path
> We are in the process of implementing a custom `FileInputFormat` that receives an Iceberg table identifier and a serialized expression `ExprNodeDesc` as input. This is similar in a lot of ways to what you can find in the `PigParquetReader` class in the `iceberg-pig` package or in `HiveHBaseTableInputFormat` class in Hive.
> 
> 
> Write path
> We have made less progress in that front but we see a path forward by implementing a custom `OutputFormat` that would keep track of the files that are being written and gather statistics. Then, each task can dump this information on HDFS. From there, the final Hive `MoveTask` can merge those “pre-manifest” files to create a new snapshot and commit the new version of a table.
> 
> 
> We hope that our observations will start a healthy conversation about supporting Iceberg tables in Hive :)
> 
> 
> Cheers,
> Adrien

Re: [DISCUSS] Implementation strategies for supporting Iceberg tables in Hive

[Ryan Blue <rb...@netflix.com>]

> With an iceberg raw store, I suspect that you might not need a storage
handler and could go straight to a input/output format. You would probably
need an input and output format for each of the storage formats:
Iceberg{Orc,Parquet,Avro}{Input,Output}Format.

I don't think that would work because Iceberg tables can use different
storage formats within the same table or partition. I think an InputFormat
would need to support all file formats.

On Wed, Aug 7, 2019 at 10:49 PM Owen O'Malley <ow...@gmail.com>
wrote:

>
>
> > On Jul 24, 2019, at 22:52, Adrien Guillo <ad...@airbnb.com.invalid>
> wrote:
> >
> > Hi Iceberg folks,
> >
> > In the last few months, we (the data infrastructure team at Airbnb) have
> been closely following the project. We are currently evaluating potential
> strategies to migrate our data warehouse to Iceberg. However, we have a
> very large Hive deployment, which means we can’t really do so without
> support for Iceberg tables in Hive.
> >
> > We have been thinking about implementation strategies. Here are some
> thoughts that we would like to share them with you:
> >
> > – Implementing a new `RawStore`
>
> My thought would be to use the embedded metastore with an iceberg
> rawstore. That enables the client to do the work rather than pushing it to
> an external metastore.
>
> I expect that some new users would be able to just use iceberg as their
> only metastore, but that others would want to have some databases in hive
> layout and others in iceberg. We could use a delegating raw store for that.
>
> > This is something that has been mentioned several times on the mailing
> list and seems to indicate that adding support for Iceberg tables in Hive
> could be achieved without client-side modifications. Does that mean that
> the Metastore is the only process manipulating Iceberg metadata (snapshots,
> manifests)? Does that mean that for instance the `listPartition*` calls to
> the Metastore return the DataFiles associated with each partition? Per our
> understanding, it seems that supporting Iceberg tables in Hive with this
> strategy will most likely require to update the RawStore interface AND will
> require at least some client-side changes. In addition, with this strategy
> the Metastore bears new responsibilities, which contradicts one of the
> Iceberg design goals: offloading more work to jobs and removing the
> metastore as a bottleneck. In the Iceberg world, not much is needed from
> the Metastore: it just keeps track of the metadata location and provides a
> mechanism for atomically updating this location (basically, what is done in
> the `HiveTableOperations` class). We would like to design a solution that
> relies  as little as possible on the Metastore so that in future we have
> the option to replace our fleet of Metastores with a simpler system.
> >
> >
> > – Implementing a new `HiveStorageHandler`
>
> With an iceberg raw store, I suspect that you might not need a storage
> handler and could go straight to a input/output format. You would probably
> need an input and output format for each of the storage formats:
> Iceberg{Orc,Parquet,Avro}{Input,Output}Format.
>
> .. Owen
> >
> > We are working on implementing custom `InputFormat` and `OutputFormat`
> classes for Iceberg (more on that in the next paragraph) and they would fit
> in nicely with the `HiveStorageHandler` and `HiveStoragePredicateHandler`
> interfaces. However, the `HiveMetaHook` interface does not seem rich enough
> to accommodate all the workflows, for instance no hooks run on `ALTER ...`
> or `INSERT...` commands.
> >
> >
> >
> > – Proof of concept
> >
> > We set out to write a proof of concept that would allow us to learn and
> experiment. We based our work on the 2.3 branch. Here’s the state of the
> project and the paths we explored:
> >
> > DDL commands
> > We support some commands such as `CREABLE TABLE ...`, `DESC ...`, `SHOW
> PARTITIONS`. They are all implemented in the client and mostly rely on the
> `HiveCatalog` class to do the work.
> >
> > Read path
> > We are in the process of implementing a custom `FileInputFormat` that
> receives an Iceberg table identifier and a serialized expression
> `ExprNodeDesc` as input. This is similar in a lot of ways to what you can
> find in the `PigParquetReader` class in the `iceberg-pig` package or in
> `HiveHBaseTableInputFormat` class in Hive.
> >
> >
> > Write path
> > We have made less progress in that front but we see a path forward by
> implementing a custom `OutputFormat` that would keep track of the files
> that are being written and gather statistics. Then, each task can dump this
> information on HDFS. From there, the final Hive `MoveTask` can merge those
> “pre-manifest” files to create a new snapshot and commit the new version of
> a table.
> >
> >
> > We hope that our observations will start a healthy conversation about
> supporting Iceberg tables in Hive :)
> >
> >
> > Cheers,
> > Adrien
>


-- 
Ryan Blue
Software Engineer
Netflix