You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iceberg.apache.org by Marko Babic <ma...@narrative.io.INVALID> on 2020/11/19 18:07:13 UTC
Integrating Existing Iceberg Tables with a Metastore
Hi everyone,
At my org we’ve spun up a few Iceberg tables on top of S3 without a
metastore (conscious of the consequences) and we’ve arrived at the point
that we need to support concurrent writes. :) I was hoping to get some
advice as to what the best way to integrate an existing Iceberg table into
a Hive Metastore or an alternative might be. We’re still relatively early
in our adoption of Iceberg and have no real prior experience with Hive so I
don’t know what I don’t know.
Some options we’re weighing:
- Existing tables aren’t so big that the moral equivalent of "CREATE
TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the
question, but we’d prefer to not have to read + rewrite everything. We also
have stateful readers (tracking which snapshots they have previously read)
and preserving table history would make life easier.
- Doing something along the lines of the following and importing the
tables into Hive as external tables looks like it should work given my
understanding of how Iceberg is using HMS, but I don’t know if it’s
encouraged and I haven’t done diligence to understand potential
consequences:
```
hive> CREATE EXTERNAL TABLE `existing_table` (...)
LOCATION
's3://existing-table/'
-- serde, input/output formats omitted
TBLPROPERTIES (
-- Assuming latest metadata file for Hadoop table is v99.metadata.json,
rename it to 00099-uuid.metadata.json
-- so that BaseMetastoreTableOperations can correctly parse the version
number.
'metadata_location'='s3://existing-table/metadata/00099-uuid.metadata.json',
'table_type'='ICEBERG'
)
```
- Others seem to have had success implementing + maintaining a custom
catalog (https://iceberg.apache.org/custom-catalog/) backed by e.g.
DynamoDB for atomic metadata updates, which could appeal to us. Seems like
migration in this case consists of implementing the catalog and plopping
the latest metadata into the backing store. Are custom catalogs more of an
escape hatch when HMS can’t be used, or would that maybe be a reasonable
way forward if we find we don’t want to maintain + operate on top of HMS?
Apologies if this was discussed or documented somewhere else and I’ve
missed it.
Thanks!
Marko
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Great to hear you're up and running, Marko!
Would you be interested in sharing your JDBC/Posgres metastore? I don't
think we have one yet and it would be great to have a simple one that is
backed by a database.
On Wed, Nov 25, 2020 at 9:06 AM Marko Babic <ma...@narrative.io.invalid>
wrote:
> Thanks for your help + suggestions, Peter. Thanks for the pointer to
> Nessie, Jacques.
>
> To wrap up the thread: I took an afternoon to put together a
> Postgres-backed metastore to make sure I understood all the moving pieces
> and have some confidence that I know what would go into the migration no
> matter what we choose (HMS vs. something custom vs. something else), just
> an internal question as to what we prefer now.
>
> Building a custom metastore was pleasantly pain-free — thanks to everyone
> involved in documenting the process + building things to have points of
> extension.
>
> Marko
>
> On Mon, Nov 23, 2020 at 12:43 AM Peter Vary <pv...@cloudera.com.invalid>
> wrote:
>
>> Hi Marco,
>>
>> See my comments below:
>>
>> On Nov 20, 2020, at 19:58, Marko Babic <ma...@narrative.io.INVALID>
>> wrote:
>>
>> Hi Peter. Thanks for responding.
>>
>> > The command you mention below: `CREATE EXTERNAL TABLE` above an
>> existing Iceberg table will not transfer the "responsibility" of tracking
>> the snapshot to HMS. It only creates a HMS external table ...
>>
>> So my understanding is that the HiveCatalog is basically just using HMS
>> as an atomically updateable pointer to a metadata file (excepting recent
>> work to make Iceberg tables queryable _from_ Hive which we won't be doing)
>>
>>
>> If you really do not want to read the tables from Hive and do not have an
>> already working HMS then Jacque's suggestion would make some sense to use
>> Nessie or another more lightweight custom catalog implementation. Maturity
>> should be considered as a plus for HMS, but probably you can get away
>> without introducing another component to your architecture.
>>
>> so what I'm doing with that command is mimicking the DDL for a
>> HiveCatalog-created table, which sets up Iceberg tables as external
>> tables in HMS
>> <https://github.com/apache/iceberg/blob/f8c68ebcb4e35db5d7f5ccb8e20d53df3abdf8b1/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L230-L292>, and
>> manually updating the `metadata_location` table property to point to latest
>> metadata file for the existing table that I want to integrate with HMS.
>> Updating the metadata pointer, along with obviously updating all
>> readers/writers to load the table via the HiveCatalog, seems to be all I
>> need to do to make that work but I'm just naively dipping my toes in here
>> and could absolutely be missing something. E.g. I figured out I'd have to
>> rename the latest metadata file from the existing table I want to integrate
>> with HMS so that BaseMetastoreTableOperations could parse the version
>> number, but only realized later that I'd have to rename _all_ the old
>> metadata files + twiddle the metadata log entries to use the updated names.
>>
>>
>> I would try to avoid renaming files, because then you get yourself into
>> the mess when you are updating all the path information in the metadata
>> files, which with the snapshot/manifest hierarchy might cause headaches.
>> AFAIK there is no requirements for the naming of the snapshot files for
>> the HiveCatalog as it is using UUID to make sure the name of the snapshot
>> files are unique. If I know correctly the v1/v2 etc requirement is only for
>> HadoopCatalog and HadoopTables.
>> So you can try just manually updating the metadata location. This could
>> work, but I do not know of any official tool for this.
>>
>>
>> > What I would do is this: ...
>>
>> Makes sense, the external table creation + metadata pointer mangling is
>> my attempt to do basically this but I'm not confident I know everything
>> that needs to go into making step 2 happen. :)
>>
>> The following is what I'm thinking:
>>
>> - Given an existing Hadoop table on top of S3 at s3://old_table/, create
>> a new table with the same schema + partition spec via HiveCatalog.
>> - Parse metadata files from the old table and update them to be
>> HadoopCatalog-compatible: all I'd be updating is metadata file names + the
>> metadata log as described above.
>> - Write updated metadata files to s3://old_table/metadata/. Update new
>> table in HMS to point to latest, updated metadata file and update the table
>> location to point to s3://old_table/.
>>
>> I could alternatively `aws s3 sync` data files from the old table to the
>> new one, rewrite all the old metadata + snapshot manifest lists + manifest
>> files to point to the new data directory, and leave s3://old_table/
>> untouched, but I guess that's a decision I'd make once I'm into things and
>> have a better sense of what'd be less error-prone.
>>
>> Thanks again!
>>
>> Marko
>>
>>
>> On Fri, Nov 20, 2020 at 12:39 AM Peter Vary <pv...@cloudera.com.invalid>
>> wrote:
>>
>>> Hi Marko,
>>>
>>> The command you mention below: `CREATE EXTERNAL TABLE` above an existing
>>> Iceberg table will not transfer the "responsibility" of tracking the
>>> snapshot to HMS. It only creates a HMS external table which will allow Hive
>>> queries to read the given table. If you want to track the snapshot in the
>>> HMS then you have to originally create a table in HMS using HiveCatalog.
>>>
>>> What I would do is this:
>>>
>>> 1. Create a new Iceberg table in a catalog which supports concurrent
>>> writes (Hive/Hadoop/Custom)
>>> 2. Migrate the tables to the new catalog. Maybe there are some
>>> already existing tools there, or with some java/spark code the snapshot
>>> files can be read and rewritten. By my understanding you definitely do not
>>> have to rewrite the data files, just the snapshot files (and maybe the
>>> manifest files)
>>>
>>> Hope this helps,
>>> Peter
>>>
>>>
>>> On Nov 19, 2020, at 21:29, John Clara <jo...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> My team has been using the custom catalog along with atomic metadata
>>> updates but we never migrated existing iceberg tables onto it. We also
>>> haven't turned on integration with the hive catalog, so I'm not sure how
>>>
>>> easy it is to plug in there (I think there was some recent work on
>>> that?). Dynamo provides a local mock which you could combine with s3mock
>>> (check iceberg tests) to test it out:
>>> https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
>>>
>>>
>>> Only weird things we've run into with dynamo is:
>>> 1. it seems like we get rate limited by dynamo pretty hard when first
>>> writing to a new table until rate limits are adjusted (potentially by aws
>>> dynamically dynamo's internal partitions?)
>>> 2. make sure to page scans if you have a lot of values when doing lists
>>> (we haven't enabled catalog listing yet, but we've ran into this before)
>>>
>>> We chose dynamo because we were using it for other usecases. I'm not
>>> sure if it's the best aws provided option for atomic changes.
>>>
>>> John
>>>
>>> On 11/19/20 10:07 AM, Marko Babic wrote:
>>>
>>> Hi everyone,
>>>
>>> At my org we’ve spun up a few Iceberg tables on top of S3 without a
>>> metastore (conscious of the consequences) and we’ve arrived at the point
>>> that we need to support concurrent writes. :) I was hoping to get some
>>> advice as to what the best way to integrate an existing Iceberg table into
>>> a Hive Metastore or an alternative might be. We’re still relatively early
>>> in our adoption of Iceberg and have no real prior experience with Hive so I
>>> don’t know what I don’t know.
>>>
>>> Some options we’re weighing:
>>>
>>> - Existing tables aren’t so big that the moral equivalent of "CREATE
>>> TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the
>>> question, but we’d prefer to not have to read + rewrite everything. We also
>>> have stateful readers (tracking which snapshots they have previously read)
>>> and preserving table history would make life easier.
>>>
>>> - Doing something along the lines of the following and importing the
>>> tables into Hive as external tables looks like it should work given my
>>> understanding of how Iceberg is using HMS, but I don’t know if it’s
>>> encouraged and I haven’t done diligence to understand potential
>>> consequences:
>>>
>>> ```
>>> hive> CREATE EXTERNAL TABLE `existing_table` (...)
>>> LOCATION
>>> 's3://existing-table/'
>>> -- serde, input/output formats omitted
>>> TBLPROPERTIES (
>>> -- Assuming latest metadata file for Hadoop table is
>>> v99.metadata.json, rename it to 00099-uuid.metadata.json
>>> -- so that BaseMetastoreTableOperations can correctly parse the
>>> version number.
>>> 'metadata_location'='
>>> s3://existing-table/metadata/00099-uuid.metadata.json',
>>> 'table_type'='ICEBERG'
>>> )
>>> ```
>>>
>>> - Others seem to have had success implementing + maintaining a custom
>>> catalog (https://iceberg.apache.org/custom-catalog/ <
>>> https://iceberg.apache.org/custom-catalog/>) backed by e.g. DynamoDB
>>> for atomic metadata updates, which could appeal to us. Seems like migration
>>> in this case consists of implementing the catalog and plopping the latest
>>> metadata into the backing store. Are custom catalogs more of an escape
>>> hatch when HMS can’t be used, or would that maybe be a reasonable way
>>> forward if we find we don’t want to maintain + operate on top of HMS?
>>>
>>> Apologies if this was discussed or documented somewhere else and I’ve
>>> missed it.
>>>
>>> Thanks!
>>>
>>> Marko
>>>
>>>
>>>
>>
--
Ryan Blue
Software Engineer
Netflix
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by Marko Babic <ma...@narrative.io.INVALID>.
Thanks for your help + suggestions, Peter. Thanks for the pointer to
Nessie, Jacques.
To wrap up the thread: I took an afternoon to put together a
Postgres-backed metastore to make sure I understood all the moving pieces
and have some confidence that I know what would go into the migration no
matter what we choose (HMS vs. something custom vs. something else), just
an internal question as to what we prefer now.
Building a custom metastore was pleasantly pain-free — thanks to everyone
involved in documenting the process + building things to have points of
extension.
Marko
On Mon, Nov 23, 2020 at 12:43 AM Peter Vary <pv...@cloudera.com.invalid>
wrote:
> Hi Marco,
>
> See my comments below:
>
> On Nov 20, 2020, at 19:58, Marko Babic <ma...@narrative.io.INVALID> wrote:
>
> Hi Peter. Thanks for responding.
>
> > The command you mention below: `CREATE EXTERNAL TABLE` above an existing
> Iceberg table will not transfer the "responsibility" of tracking the
> snapshot to HMS. It only creates a HMS external table ...
>
> So my understanding is that the HiveCatalog is basically just using HMS as
> an atomically updateable pointer to a metadata file (excepting recent work
> to make Iceberg tables queryable _from_ Hive which we won't be doing)
>
>
> If you really do not want to read the tables from Hive and do not have an
> already working HMS then Jacque's suggestion would make some sense to use
> Nessie or another more lightweight custom catalog implementation. Maturity
> should be considered as a plus for HMS, but probably you can get away
> without introducing another component to your architecture.
>
> so what I'm doing with that command is mimicking the DDL for a
> HiveCatalog-created table, which sets up Iceberg tables as external
> tables in HMS
> <https://github.com/apache/iceberg/blob/f8c68ebcb4e35db5d7f5ccb8e20d53df3abdf8b1/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L230-L292>, and
> manually updating the `metadata_location` table property to point to latest
> metadata file for the existing table that I want to integrate with HMS.
> Updating the metadata pointer, along with obviously updating all
> readers/writers to load the table via the HiveCatalog, seems to be all I
> need to do to make that work but I'm just naively dipping my toes in here
> and could absolutely be missing something. E.g. I figured out I'd have to
> rename the latest metadata file from the existing table I want to integrate
> with HMS so that BaseMetastoreTableOperations could parse the version
> number, but only realized later that I'd have to rename _all_ the old
> metadata files + twiddle the metadata log entries to use the updated names.
>
>
> I would try to avoid renaming files, because then you get yourself into
> the mess when you are updating all the path information in the metadata
> files, which with the snapshot/manifest hierarchy might cause headaches.
> AFAIK there is no requirements for the naming of the snapshot files for
> the HiveCatalog as it is using UUID to make sure the name of the snapshot
> files are unique. If I know correctly the v1/v2 etc requirement is only for
> HadoopCatalog and HadoopTables.
> So you can try just manually updating the metadata location. This could
> work, but I do not know of any official tool for this.
>
>
> > What I would do is this: ...
>
> Makes sense, the external table creation + metadata pointer mangling is my
> attempt to do basically this but I'm not confident I know everything that
> needs to go into making step 2 happen. :)
>
> The following is what I'm thinking:
>
> - Given an existing Hadoop table on top of S3 at s3://old_table/, create
> a new table with the same schema + partition spec via HiveCatalog.
> - Parse metadata files from the old table and update them to be
> HadoopCatalog-compatible: all I'd be updating is metadata file names + the
> metadata log as described above.
> - Write updated metadata files to s3://old_table/metadata/. Update new
> table in HMS to point to latest, updated metadata file and update the table
> location to point to s3://old_table/.
>
> I could alternatively `aws s3 sync` data files from the old table to the
> new one, rewrite all the old metadata + snapshot manifest lists + manifest
> files to point to the new data directory, and leave s3://old_table/
> untouched, but I guess that's a decision I'd make once I'm into things and
> have a better sense of what'd be less error-prone.
>
> Thanks again!
>
> Marko
>
>
> On Fri, Nov 20, 2020 at 12:39 AM Peter Vary <pv...@cloudera.com.invalid>
> wrote:
>
>> Hi Marko,
>>
>> The command you mention below: `CREATE EXTERNAL TABLE` above an existing
>> Iceberg table will not transfer the "responsibility" of tracking the
>> snapshot to HMS. It only creates a HMS external table which will allow Hive
>> queries to read the given table. If you want to track the snapshot in the
>> HMS then you have to originally create a table in HMS using HiveCatalog.
>>
>> What I would do is this:
>>
>> 1. Create a new Iceberg table in a catalog which supports concurrent
>> writes (Hive/Hadoop/Custom)
>> 2. Migrate the tables to the new catalog. Maybe there are some
>> already existing tools there, or with some java/spark code the snapshot
>> files can be read and rewritten. By my understanding you definitely do not
>> have to rewrite the data files, just the snapshot files (and maybe the
>> manifest files)
>>
>> Hope this helps,
>> Peter
>>
>>
>> On Nov 19, 2020, at 21:29, John Clara <jo...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> My team has been using the custom catalog along with atomic metadata
>> updates but we never migrated existing iceberg tables onto it. We also
>> haven't turned on integration with the hive catalog, so I'm not sure how
>>
>> easy it is to plug in there (I think there was some recent work on
>> that?). Dynamo provides a local mock which you could combine with s3mock
>> (check iceberg tests) to test it out:
>> https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
>>
>>
>> Only weird things we've run into with dynamo is:
>> 1. it seems like we get rate limited by dynamo pretty hard when first
>> writing to a new table until rate limits are adjusted (potentially by aws
>> dynamically dynamo's internal partitions?)
>> 2. make sure to page scans if you have a lot of values when doing lists
>> (we haven't enabled catalog listing yet, but we've ran into this before)
>>
>> We chose dynamo because we were using it for other usecases. I'm not sure
>> if it's the best aws provided option for atomic changes.
>>
>> John
>>
>> On 11/19/20 10:07 AM, Marko Babic wrote:
>>
>> Hi everyone,
>>
>> At my org we’ve spun up a few Iceberg tables on top of S3 without a
>> metastore (conscious of the consequences) and we’ve arrived at the point
>> that we need to support concurrent writes. :) I was hoping to get some
>> advice as to what the best way to integrate an existing Iceberg table into
>> a Hive Metastore or an alternative might be. We’re still relatively early
>> in our adoption of Iceberg and have no real prior experience with Hive so I
>> don’t know what I don’t know.
>>
>> Some options we’re weighing:
>>
>> - Existing tables aren’t so big that the moral equivalent of "CREATE
>> TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the
>> question, but we’d prefer to not have to read + rewrite everything. We also
>> have stateful readers (tracking which snapshots they have previously read)
>> and preserving table history would make life easier.
>>
>> - Doing something along the lines of the following and importing the
>> tables into Hive as external tables looks like it should work given my
>> understanding of how Iceberg is using HMS, but I don’t know if it’s
>> encouraged and I haven’t done diligence to understand potential
>> consequences:
>>
>> ```
>> hive> CREATE EXTERNAL TABLE `existing_table` (...)
>> LOCATION
>> 's3://existing-table/'
>> -- serde, input/output formats omitted
>> TBLPROPERTIES (
>> -- Assuming latest metadata file for Hadoop table is
>> v99.metadata.json, rename it to 00099-uuid.metadata.json
>> -- so that BaseMetastoreTableOperations can correctly parse the
>> version number.
>> 'metadata_location'='
>> s3://existing-table/metadata/00099-uuid.metadata.json',
>> 'table_type'='ICEBERG'
>> )
>> ```
>>
>> - Others seem to have had success implementing + maintaining a custom
>> catalog (https://iceberg.apache.org/custom-catalog/ <
>> https://iceberg.apache.org/custom-catalog/>) backed by e.g. DynamoDB for
>> atomic metadata updates, which could appeal to us. Seems like migration in
>> this case consists of implementing the catalog and plopping the latest
>> metadata into the backing store. Are custom catalogs more of an escape
>> hatch when HMS can’t be used, or would that maybe be a reasonable way
>> forward if we find we don’t want to maintain + operate on top of HMS?
>>
>> Apologies if this was discussed or documented somewhere else and I’ve
>> missed it.
>>
>> Thanks!
>>
>> Marko
>>
>>
>>
>
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by Peter Vary <pv...@cloudera.com.INVALID>.
Hi Marco,
See my comments below:
> On Nov 20, 2020, at 19:58, Marko Babic <ma...@narrative.io.INVALID> wrote:
>
> Hi Peter. Thanks for responding.
>
> > The command you mention below: `CREATE EXTERNAL TABLE` above an existing Iceberg table will not transfer the "responsibility" of tracking the snapshot to HMS. It only creates a HMS external table ...
>
> So my understanding is that the HiveCatalog is basically just using HMS as an atomically updateable pointer to a metadata file (excepting recent work to make Iceberg tables queryable _from_ Hive which we won't be doing)
If you really do not want to read the tables from Hive and do not have an already working HMS then Jacque's suggestion would make some sense to use Nessie or another more lightweight custom catalog implementation. Maturity should be considered as a plus for HMS, but probably you can get away without introducing another component to your architecture.
> so what I'm doing with that command is mimicking the DDL for a HiveCatalog-created table, which sets up Iceberg tables as external tables in HMS <https://github.com/apache/iceberg/blob/f8c68ebcb4e35db5d7f5ccb8e20d53df3abdf8b1/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L230-L292>, and manually updating the `metadata_location` table property to point to latest metadata file for the existing table that I want to integrate with HMS. Updating the metadata pointer, along with obviously updating all readers/writers to load the table via the HiveCatalog, seems to be all I need to do to make that work but I'm just naively dipping my toes in here and could absolutely be missing something. E.g. I figured out I'd have to rename the latest metadata file from the existing table I want to integrate with HMS so that BaseMetastoreTableOperations could parse the version number, but only realized later that I'd have to rename _all_ the old metadata files + twiddle the metadata log entries to use the updated names.
I would try to avoid renaming files, because then you get yourself into the mess when you are updating all the path information in the metadata files, which with the snapshot/manifest hierarchy might cause headaches.
AFAIK there is no requirements for the naming of the snapshot files for the HiveCatalog as it is using UUID to make sure the name of the snapshot files are unique. If I know correctly the v1/v2 etc requirement is only for HadoopCatalog and HadoopTables.
So you can try just manually updating the metadata location. This could work, but I do not know of any official tool for this.
>
> > What I would do is this: ...
>
> Makes sense, the external table creation + metadata pointer mangling is my attempt to do basically this but I'm not confident I know everything that needs to go into making step 2 happen. :)
>
> The following is what I'm thinking:
>
> - Given an existing Hadoop table on top of S3 at s3://old_table/, create a new table with the same schema + partition spec via HiveCatalog.
> - Parse metadata files from the old table and update them to be HadoopCatalog-compatible: all I'd be updating is metadata file names + the metadata log as described above.
> - Write updated metadata files to s3://old_table/metadata/. Update new table in HMS to point to latest, updated metadata file and update the table location to point to s3://old_table/.
>
> I could alternatively `aws s3 sync` data files from the old table to the new one, rewrite all the old metadata + snapshot manifest lists + manifest files to point to the new data directory, and leave s3://old_table/ untouched, but I guess that's a decision I'd make once I'm into things and have a better sense of what'd be less error-prone.
>
> Thanks again!
>
> Marko
>
>
> On Fri, Nov 20, 2020 at 12:39 AM Peter Vary <pv...@cloudera.com.invalid> wrote:
> Hi Marko,
>
> The command you mention below: `CREATE EXTERNAL TABLE` above an existing Iceberg table will not transfer the "responsibility" of tracking the snapshot to HMS. It only creates a HMS external table which will allow Hive queries to read the given table. If you want to track the snapshot in the HMS then you have to originally create a table in HMS using HiveCatalog.
>
> What I would do is this:
> Create a new Iceberg table in a catalog which supports concurrent writes (Hive/Hadoop/Custom)
> Migrate the tables to the new catalog. Maybe there are some already existing tools there, or with some java/spark code the snapshot files can be read and rewritten. By my understanding you definitely do not have to rewrite the data files, just the snapshot files (and maybe the manifest files)
> Hope this helps,
> Peter
>
>
>> On Nov 19, 2020, at 21:29, John Clara <john.anthony.clara@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi,
>>
>> My team has been using the custom catalog along with atomic metadata updates but we never migrated existing iceberg tables onto it. We also haven't turned on integration with the hive catalog, so I'm not sure how
>>
>> easy it is to plug in there (I think there was some recent work on that?). Dynamo provides a local mock which you could combine with s3mock (check iceberg tests) to test it out: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html <https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html>
>>
>> Only weird things we've run into with dynamo is:
>> 1. it seems like we get rate limited by dynamo pretty hard when first writing to a new table until rate limits are adjusted (potentially by aws dynamically dynamo's internal partitions?)
>> 2. make sure to page scans if you have a lot of values when doing lists (we haven't enabled catalog listing yet, but we've ran into this before)
>>
>> We chose dynamo because we were using it for other usecases. I'm not sure if it's the best aws provided option for atomic changes.
>>
>> John
>>
>> On 11/19/20 10:07 AM, Marko Babic wrote:
>>> Hi everyone,
>>>
>>> At my org we’ve spun up a few Iceberg tables on top of S3 without a metastore (conscious of the consequences) and we’ve arrived at the point that we need to support concurrent writes. :) I was hoping to get some advice as to what the best way to integrate an existing Iceberg table into a Hive Metastore or an alternative might be. We’re still relatively early in our adoption of Iceberg and have no real prior experience with Hive so I don’t know what I don’t know.
>>>
>>> Some options we’re weighing:
>>>
>>> - Existing tables aren’t so big that the moral equivalent of "CREATE TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the question, but we’d prefer to not have to read + rewrite everything. We also have stateful readers (tracking which snapshots they have previously read) and preserving table history would make life easier.
>>>
>>> - Doing something along the lines of the following and importing the tables into Hive as external tables looks like it should work given my understanding of how Iceberg is using HMS, but I don’t know if it’s encouraged and I haven’t done diligence to understand potential consequences:
>>>
>>> ```
>>> hive> CREATE EXTERNAL TABLE `existing_table` (...)
>>> LOCATION
>>> 's3://existing-table/ <>'
>>> -- serde, input/output formats omitted
>>> TBLPROPERTIES (
>>> -- Assuming latest metadata file for Hadoop table is v99.metadata.json, rename it to 00099-uuid.metadata.json
>>> -- so that BaseMetastoreTableOperations can correctly parse the version number.
>>> 'metadata_location'='s3://existing-table/metadata/00099-uuid.metadata.json <>',
>>> 'table_type'='ICEBERG'
>>> )
>>> ```
>>>
>>> - Others seem to have had success implementing + maintaining a custom catalog (https://iceberg.apache.org/custom-catalog/ <https://iceberg.apache.org/custom-catalog/> <https://iceberg.apache.org/custom-catalog/ <https://iceberg.apache.org/custom-catalog/>>) backed by e.g. DynamoDB for atomic metadata updates, which could appeal to us. Seems like migration in this case consists of implementing the catalog and plopping the latest metadata into the backing store. Are custom catalogs more of an escape hatch when HMS can’t be used, or would that maybe be a reasonable way forward if we find we don’t want to maintain + operate on top of HMS?
>>>
>>> Apologies if this was discussed or documented somewhere else and I’ve missed it.
>>>
>>> Thanks!
>>>
>>> Marko
>
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by Jacques Nadeau <ja...@dremio.com>.
FYI, I would avoid adopting HMS because you need a better catalog. While
the HMS Iceberg catalog is mature, you're adopting something (HMS) that
carries a lot of baggage. I'd look at the other catalogs that are up and
coming if you can.
For example, Nessie (projectnessie.org) was built to provide a cloud native
approach to Iceberg transaction arbitration (along with some other nifty
features around cross-table transactions and git semantics) so that people
who work in the cloud but don't use Hive metastore, don't have to start.
The HA complexity, scaling dynamics and overall operational load of Nessie
is targeted to be a fraction of what HMS is.
Full disclosure, I work on Nessie.
Foot for thought, anyway.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Fri, Nov 20, 2020 at 10:58 AM Marko Babic <ma...@narrative.io.invalid>
wrote:
> Hi Peter. Thanks for responding.
>
> > The command you mention below: `CREATE EXTERNAL TABLE` above an existing
> Iceberg table will not transfer the "responsibility" of tracking the
> snapshot to HMS. It only creates a HMS external table ...
>
> So my understanding is that the HiveCatalog is basically just using HMS as
> an atomically updateable pointer to a metadata file (excepting recent work
> to make Iceberg tables queryable _from_ Hive which we won't be doing) so
> what I'm doing with that command is mimicking the DDL for a
> HiveCatalog-created table, which sets up Iceberg tables as external
> tables in HMS
> <https://github.com/apache/iceberg/blob/f8c68ebcb4e35db5d7f5ccb8e20d53df3abdf8b1/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L230-L292>, and
> manually updating the `metadata_location` table property to point to latest
> metadata file for the existing table that I want to integrate with HMS.
> Updating the metadata pointer, along with obviously updating all
> readers/writers to load the table via the HiveCatalog, seems to be all I
> need to do to make that work but I'm just naively dipping my toes in here
> and could absolutely be missing something. E.g. I figured out I'd have to
> rename the latest metadata file from the existing table I want to integrate
> with HMS so that BaseMetastoreTableOperations could parse the version
> number, but only realized later that I'd have to rename _all_ the old
> metadata files + twiddle the metadata log entries to use the updated names.
>
> > What I would do is this: ...
>
> Makes sense, the external table creation + metadata pointer mangling is my
> attempt to do basically this but I'm not confident I know everything that
> needs to go into making step 2 happen. :)
>
> The following is what I'm thinking:
>
> - Given an existing Hadoop table on top of S3 at s3://old_table/, create a
> new table with the same schema + partition spec via HiveCatalog.
> - Parse metadata files from the old table and update them to be
> HadoopCatalog-compatible: all I'd be updating is metadata file names + the
> metadata log as described above.
> - Write updated metadata files to s3://old_table/metadata/. Update new
> table in HMS to point to latest, updated metadata file and update the table
> location to point to s3://old_table/.
>
> I could alternatively `aws s3 sync` data files from the old table to the
> new one, rewrite all the old metadata + snapshot manifest lists + manifest
> files to point to the new data directory, and leave s3://old_table/
> untouched, but I guess that's a decision I'd make once I'm into things and
> have a better sense of what'd be less error-prone.
>
> Thanks again!
>
> Marko
>
>
> On Fri, Nov 20, 2020 at 12:39 AM Peter Vary <pv...@cloudera.com.invalid>
> wrote:
>
>> Hi Marko,
>>
>> The command you mention below: `CREATE EXTERNAL TABLE` above an existing
>> Iceberg table will not transfer the "responsibility" of tracking the
>> snapshot to HMS. It only creates a HMS external table which will allow Hive
>> queries to read the given table. If you want to track the snapshot in the
>> HMS then you have to originally create a table in HMS using HiveCatalog.
>>
>> What I would do is this:
>>
>> 1. Create a new Iceberg table in a catalog which supports concurrent
>> writes (Hive/Hadoop/Custom)
>> 2. Migrate the tables to the new catalog. Maybe there are some
>> already existing tools there, or with some java/spark code the snapshot
>> files can be read and rewritten. By my understanding you definitely do not
>> have to rewrite the data files, just the snapshot files (and maybe the
>> manifest files)
>>
>> Hope this helps,
>> Peter
>>
>>
>> On Nov 19, 2020, at 21:29, John Clara <jo...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> My team has been using the custom catalog along with atomic metadata
>> updates but we never migrated existing iceberg tables onto it. We also
>> haven't turned on integration with the hive catalog, so I'm not sure how
>>
>> easy it is to plug in there (I think there was some recent work on
>> that?). Dynamo provides a local mock which you could combine with s3mock
>> (check iceberg tests) to test it out:
>> https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
>>
>>
>> Only weird things we've run into with dynamo is:
>> 1. it seems like we get rate limited by dynamo pretty hard when first
>> writing to a new table until rate limits are adjusted (potentially by aws
>> dynamically dynamo's internal partitions?)
>> 2. make sure to page scans if you have a lot of values when doing lists
>> (we haven't enabled catalog listing yet, but we've ran into this before)
>>
>> We chose dynamo because we were using it for other usecases. I'm not sure
>> if it's the best aws provided option for atomic changes.
>>
>> John
>>
>> On 11/19/20 10:07 AM, Marko Babic wrote:
>>
>> Hi everyone,
>>
>> At my org we’ve spun up a few Iceberg tables on top of S3 without a
>> metastore (conscious of the consequences) and we’ve arrived at the point
>> that we need to support concurrent writes. :) I was hoping to get some
>> advice as to what the best way to integrate an existing Iceberg table into
>> a Hive Metastore or an alternative might be. We’re still relatively early
>> in our adoption of Iceberg and have no real prior experience with Hive so I
>> don’t know what I don’t know.
>>
>> Some options we’re weighing:
>>
>> - Existing tables aren’t so big that the moral equivalent of "CREATE
>> TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the
>> question, but we’d prefer to not have to read + rewrite everything. We also
>> have stateful readers (tracking which snapshots they have previously read)
>> and preserving table history would make life easier.
>>
>> - Doing something along the lines of the following and importing the
>> tables into Hive as external tables looks like it should work given my
>> understanding of how Iceberg is using HMS, but I don’t know if it’s
>> encouraged and I haven’t done diligence to understand potential
>> consequences:
>>
>> ```
>> hive> CREATE EXTERNAL TABLE `existing_table` (...)
>> LOCATION
>> 's3://existing-table/'
>> -- serde, input/output formats omitted
>> TBLPROPERTIES (
>> -- Assuming latest metadata file for Hadoop table is
>> v99.metadata.json, rename it to 00099-uuid.metadata.json
>> -- so that BaseMetastoreTableOperations can correctly parse the
>> version number.
>> 'metadata_location'='
>> s3://existing-table/metadata/00099-uuid.metadata.json',
>> 'table_type'='ICEBERG'
>> )
>> ```
>>
>> - Others seem to have had success implementing + maintaining a custom
>> catalog (https://iceberg.apache.org/custom-catalog/ <
>> https://iceberg.apache.org/custom-catalog/>) backed by e.g. DynamoDB for
>> atomic metadata updates, which could appeal to us. Seems like migration in
>> this case consists of implementing the catalog and plopping the latest
>> metadata into the backing store. Are custom catalogs more of an escape
>> hatch when HMS can’t be used, or would that maybe be a reasonable way
>> forward if we find we don’t want to maintain + operate on top of HMS?
>>
>> Apologies if this was discussed or documented somewhere else and I’ve
>> missed it.
>>
>> Thanks!
>>
>> Marko
>>
>>
>>
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by Marko Babic <ma...@narrative.io.INVALID>.
Hi Peter. Thanks for responding.
> The command you mention below: `CREATE EXTERNAL TABLE` above an existing
Iceberg table will not transfer the "responsibility" of tracking the
snapshot to HMS. It only creates a HMS external table ...
So my understanding is that the HiveCatalog is basically just using HMS as
an atomically updateable pointer to a metadata file (excepting recent work
to make Iceberg tables queryable _from_ Hive which we won't be doing) so
what I'm doing with that command is mimicking the DDL for a
HiveCatalog-created table, which sets up Iceberg tables as external tables
in HMS
<https://github.com/apache/iceberg/blob/f8c68ebcb4e35db5d7f5ccb8e20d53df3abdf8b1/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java#L230-L292>,
and
manually updating the `metadata_location` table property to point to latest
metadata file for the existing table that I want to integrate with HMS.
Updating the metadata pointer, along with obviously updating all
readers/writers to load the table via the HiveCatalog, seems to be all I
need to do to make that work but I'm just naively dipping my toes in here
and could absolutely be missing something. E.g. I figured out I'd have to
rename the latest metadata file from the existing table I want to integrate
with HMS so that BaseMetastoreTableOperations could parse the version
number, but only realized later that I'd have to rename _all_ the old
metadata files + twiddle the metadata log entries to use the updated names.
> What I would do is this: ...
Makes sense, the external table creation + metadata pointer mangling is my
attempt to do basically this but I'm not confident I know everything that
needs to go into making step 2 happen. :)
The following is what I'm thinking:
- Given an existing Hadoop table on top of S3 at s3://old_table/, create a
new table with the same schema + partition spec via HiveCatalog.
- Parse metadata files from the old table and update them to be
HadoopCatalog-compatible: all I'd be updating is metadata file names + the
metadata log as described above.
- Write updated metadata files to s3://old_table/metadata/. Update new
table in HMS to point to latest, updated metadata file and update the table
location to point to s3://old_table/.
I could alternatively `aws s3 sync` data files from the old table to the
new one, rewrite all the old metadata + snapshot manifest lists + manifest
files to point to the new data directory, and leave s3://old_table/
untouched, but I guess that's a decision I'd make once I'm into things and
have a better sense of what'd be less error-prone.
Thanks again!
Marko
On Fri, Nov 20, 2020 at 12:39 AM Peter Vary <pv...@cloudera.com.invalid>
wrote:
> Hi Marko,
>
> The command you mention below: `CREATE EXTERNAL TABLE` above an existing
> Iceberg table will not transfer the "responsibility" of tracking the
> snapshot to HMS. It only creates a HMS external table which will allow Hive
> queries to read the given table. If you want to track the snapshot in the
> HMS then you have to originally create a table in HMS using HiveCatalog.
>
> What I would do is this:
>
> 1. Create a new Iceberg table in a catalog which supports concurrent
> writes (Hive/Hadoop/Custom)
> 2. Migrate the tables to the new catalog. Maybe there are some already
> existing tools there, or with some java/spark code the snapshot files can
> be read and rewritten. By my understanding you definitely do not have to
> rewrite the data files, just the snapshot files (and maybe the manifest
> files)
>
> Hope this helps,
> Peter
>
>
> On Nov 19, 2020, at 21:29, John Clara <jo...@gmail.com>
> wrote:
>
> Hi,
>
> My team has been using the custom catalog along with atomic metadata
> updates but we never migrated existing iceberg tables onto it. We also
> haven't turned on integration with the hive catalog, so I'm not sure how
>
> easy it is to plug in there (I think there was some recent work on that?).
> Dynamo provides a local mock which you could combine with s3mock (check
> iceberg tests) to test it out:
> https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
>
>
> Only weird things we've run into with dynamo is:
> 1. it seems like we get rate limited by dynamo pretty hard when first
> writing to a new table until rate limits are adjusted (potentially by aws
> dynamically dynamo's internal partitions?)
> 2. make sure to page scans if you have a lot of values when doing lists
> (we haven't enabled catalog listing yet, but we've ran into this before)
>
> We chose dynamo because we were using it for other usecases. I'm not sure
> if it's the best aws provided option for atomic changes.
>
> John
>
> On 11/19/20 10:07 AM, Marko Babic wrote:
>
> Hi everyone,
>
> At my org we’ve spun up a few Iceberg tables on top of S3 without a
> metastore (conscious of the consequences) and we’ve arrived at the point
> that we need to support concurrent writes. :) I was hoping to get some
> advice as to what the best way to integrate an existing Iceberg table into
> a Hive Metastore or an alternative might be. We’re still relatively early
> in our adoption of Iceberg and have no real prior experience with Hive so I
> don’t know what I don’t know.
>
> Some options we’re weighing:
>
> - Existing tables aren’t so big that the moral equivalent of "CREATE
> TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the
> question, but we’d prefer to not have to read + rewrite everything. We also
> have stateful readers (tracking which snapshots they have previously read)
> and preserving table history would make life easier.
>
> - Doing something along the lines of the following and importing the
> tables into Hive as external tables looks like it should work given my
> understanding of how Iceberg is using HMS, but I don’t know if it’s
> encouraged and I haven’t done diligence to understand potential
> consequences:
>
> ```
> hive> CREATE EXTERNAL TABLE `existing_table` (...)
> LOCATION
> 's3://existing-table/'
> -- serde, input/output formats omitted
> TBLPROPERTIES (
> -- Assuming latest metadata file for Hadoop table is v99.metadata.json,
> rename it to 00099-uuid.metadata.json
> -- so that BaseMetastoreTableOperations can correctly parse the version
> number.
> 'metadata_location'='s3://existing-table/metadata/00099-uuid.metadata.json
> ',
> 'table_type'='ICEBERG'
> )
> ```
>
> - Others seem to have had success implementing + maintaining a custom
> catalog (https://iceberg.apache.org/custom-catalog/ <
> https://iceberg.apache.org/custom-catalog/>) backed by e.g. DynamoDB for
> atomic metadata updates, which could appeal to us. Seems like migration in
> this case consists of implementing the catalog and plopping the latest
> metadata into the backing store. Are custom catalogs more of an escape
> hatch when HMS can’t be used, or would that maybe be a reasonable way
> forward if we find we don’t want to maintain + operate on top of HMS?
>
> Apologies if this was discussed or documented somewhere else and I’ve
> missed it.
>
> Thanks!
>
> Marko
>
>
>
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by Peter Vary <pv...@cloudera.com.INVALID>.
Hi Marko,
The command you mention below: `CREATE EXTERNAL TABLE` above an existing Iceberg table will not transfer the "responsibility" of tracking the snapshot to HMS. It only creates a HMS external table which will allow Hive queries to read the given table. If you want to track the snapshot in the HMS then you have to originally create a table in HMS using HiveCatalog.
What I would do is this:
Create a new Iceberg table in a catalog which supports concurrent writes (Hive/Hadoop/Custom)
Migrate the tables to the new catalog. Maybe there are some already existing tools there, or with some java/spark code the snapshot files can be read and rewritten. By my understanding you definitely do not have to rewrite the data files, just the snapshot files (and maybe the manifest files)
Hope this helps,
Peter
> On Nov 19, 2020, at 21:29, John Clara <jo...@gmail.com> wrote:
>
> Hi,
>
> My team has been using the custom catalog along with atomic metadata updates but we never migrated existing iceberg tables onto it. We also haven't turned on integration with the hive catalog, so I'm not sure how
>
> easy it is to plug in there (I think there was some recent work on that?). Dynamo provides a local mock which you could combine with s3mock (check iceberg tests) to test it out: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html <https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html>
>
> Only weird things we've run into with dynamo is:
> 1. it seems like we get rate limited by dynamo pretty hard when first writing to a new table until rate limits are adjusted (potentially by aws dynamically dynamo's internal partitions?)
> 2. make sure to page scans if you have a lot of values when doing lists (we haven't enabled catalog listing yet, but we've ran into this before)
>
> We chose dynamo because we were using it for other usecases. I'm not sure if it's the best aws provided option for atomic changes.
>
> John
>
> On 11/19/20 10:07 AM, Marko Babic wrote:
>> Hi everyone,
>>
>> At my org we’ve spun up a few Iceberg tables on top of S3 without a metastore (conscious of the consequences) and we’ve arrived at the point that we need to support concurrent writes. :) I was hoping to get some advice as to what the best way to integrate an existing Iceberg table into a Hive Metastore or an alternative might be. We’re still relatively early in our adoption of Iceberg and have no real prior experience with Hive so I don’t know what I don’t know.
>>
>> Some options we’re weighing:
>>
>> - Existing tables aren’t so big that the moral equivalent of "CREATE TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the question, but we’d prefer to not have to read + rewrite everything. We also have stateful readers (tracking which snapshots they have previously read) and preserving table history would make life easier.
>>
>> - Doing something along the lines of the following and importing the tables into Hive as external tables looks like it should work given my understanding of how Iceberg is using HMS, but I don’t know if it’s encouraged and I haven’t done diligence to understand potential consequences:
>>
>> ```
>> hive> CREATE EXTERNAL TABLE `existing_table` (...)
>> LOCATION
>> 's3://existing-table/'
>> -- serde, input/output formats omitted
>> TBLPROPERTIES (
>> -- Assuming latest metadata file for Hadoop table is v99.metadata.json, rename it to 00099-uuid.metadata.json
>> -- so that BaseMetastoreTableOperations can correctly parse the version number.
>> 'metadata_location'='s3://existing-table/metadata/00099-uuid.metadata.json',
>> 'table_type'='ICEBERG'
>> )
>> ```
>>
>> - Others seem to have had success implementing + maintaining a custom catalog (https://iceberg.apache.org/custom-catalog/ <https://iceberg.apache.org/custom-catalog/> <https://iceberg.apache.org/custom-catalog/ <https://iceberg.apache.org/custom-catalog/>>) backed by e.g. DynamoDB for atomic metadata updates, which could appeal to us. Seems like migration in this case consists of implementing the catalog and plopping the latest metadata into the backing store. Are custom catalogs more of an escape hatch when HMS can’t be used, or would that maybe be a reasonable way forward if we find we don’t want to maintain + operate on top of HMS?
>>
>> Apologies if this was discussed or documented somewhere else and I’ve missed it.
>>
>> Thanks!
>>
>> Marko
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by Marko Babic <ma...@narrative.io.INVALID>.
Hi John,
Thanks for the experience report and pointers to resources. :) If we do end
up going down that road it'll be super helpful.
Marko
On Thu, Nov 19, 2020 at 12:29 PM John Clara <jo...@gmail.com>
wrote:
> Hi,
>
> My team has been using the custom catalog along with atomic metadata
> updates but we never migrated existing iceberg tables onto it. We also
> haven't turned on integration with the hive catalog, so I'm not sure how
>
> easy it is to plug in there (I think there was some recent work on
> that?). Dynamo provides a local mock which you could combine with s3mock
> (check iceberg tests) to test it out:
>
> https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
>
>
> Only weird things we've run into with dynamo is:
> 1. it seems like we get rate limited by dynamo pretty hard when first
> writing to a new table until rate limits are adjusted (potentially by
> aws dynamically dynamo's internal partitions?)
> 2. make sure to page scans if you have a lot of values when doing lists
> (we haven't enabled catalog listing yet, but we've ran into this before)
>
> We chose dynamo because we were using it for other usecases. I'm not
> sure if it's the best aws provided option for atomic changes.
>
> John
>
> On 11/19/20 10:07 AM, Marko Babic wrote:
> > Hi everyone,
> >
> > At my org we’ve spun up a few Iceberg tables on top of S3 without a
> > metastore (conscious of the consequences) and we’ve arrived at the
> > point that we need to support concurrent writes. :) I was hoping to
> > get some advice as to what the best way to integrate an existing
> > Iceberg table into a Hive Metastore or an alternative might be. We’re
> > still relatively early in our adoption of Iceberg and have no real
> > prior experience with Hive so I don’t know what I don’t know.
> >
> > Some options we’re weighing:
> >
> > - Existing tables aren’t so big that the moral equivalent of "CREATE
> > TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the
> > question, but we’d prefer to not have to read + rewrite everything. We
> > also have stateful readers (tracking which snapshots they have
> > previously read) and preserving table history would make life easier.
> >
> > - Doing something along the lines of the following and importing the
> > tables into Hive as external tables looks like it should work given my
> > understanding of how Iceberg is using HMS, but I don’t know if it’s
> > encouraged and I haven’t done diligence to understand potential
> > consequences:
> >
> > ```
> > hive> CREATE EXTERNAL TABLE `existing_table` (...)
> > LOCATION
> > 's3://existing-table/'
> > -- serde, input/output formats omitted
> > TBLPROPERTIES (
> > -- Assuming latest metadata file for Hadoop table is
> > v99.metadata.json, rename it to 00099-uuid.metadata.json
> > -- so that BaseMetastoreTableOperations can correctly parse the
> > version number.
> >
> 'metadata_location'='s3://existing-table/metadata/00099-uuid.metadata.json',
> > 'table_type'='ICEBERG'
> > )
> > ```
> >
> > - Others seem to have had success implementing + maintaining a
> > custom catalog (https://iceberg.apache.org/custom-catalog/
> > <https://iceberg.apache.org/custom-catalog/>) backed by e.g. DynamoDB
> > for atomic metadata updates, which could appeal to us. Seems like
> > migration in this case consists of implementing the catalog and
> > plopping the latest metadata into the backing store. Are custom
> > catalogs more of an escape hatch when HMS can’t be used, or would that
> > maybe be a reasonable way forward if we find we don’t want to maintain
> > + operate on top of HMS?
> >
> > Apologies if this was discussed or documented somewhere else and I’ve
> > missed it.
> >
> > Thanks!
> >
> > Marko
>
Re: Integrating Existing Iceberg Tables with a Metastore
Posted by John Clara <jo...@gmail.com>.
Hi,
My team has been using the custom catalog along with atomic metadata
updates but we never migrated existing iceberg tables onto it. We also
haven't turned on integration with the hive catalog, so I'm not sure how
easy it is to plug in there (I think there was some recent work on
that?). Dynamo provides a local mock which you could combine with s3mock
(check iceberg tests) to test it out:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html
Only weird things we've run into with dynamo is:
1. it seems like we get rate limited by dynamo pretty hard when first
writing to a new table until rate limits are adjusted (potentially by
aws dynamically dynamo's internal partitions?)
2. make sure to page scans if you have a lot of values when doing lists
(we haven't enabled catalog listing yet, but we've ran into this before)
We chose dynamo because we were using it for other usecases. I'm not
sure if it's the best aws provided option for atomic changes.
John
On 11/19/20 10:07 AM, Marko Babic wrote:
> Hi everyone,
>
> At my org we’ve spun up a few Iceberg tables on top of S3 without a
> metastore (conscious of the consequences) and we’ve arrived at the
> point that we need to support concurrent writes. :) I was hoping to
> get some advice as to what the best way to integrate an existing
> Iceberg table into a Hive Metastore or an alternative might be. We’re
> still relatively early in our adoption of Iceberg and have no real
> prior experience with Hive so I don’t know what I don’t know.
>
> Some options we’re weighing:
>
> - Existing tables aren’t so big that the moral equivalent of "CREATE
> TABLE hive.db.table … AS SELECT * FROM hadoop.table" is out of the
> question, but we’d prefer to not have to read + rewrite everything. We
> also have stateful readers (tracking which snapshots they have
> previously read) and preserving table history would make life easier.
>
> - Doing something along the lines of the following and importing the
> tables into Hive as external tables looks like it should work given my
> understanding of how Iceberg is using HMS, but I don’t know if it’s
> encouraged and I haven’t done diligence to understand potential
> consequences:
>
> ```
> hive> CREATE EXTERNAL TABLE `existing_table` (...)
> LOCATION
> 's3://existing-table/'
> -- serde, input/output formats omitted
> TBLPROPERTIES (
> -- Assuming latest metadata file for Hadoop table is
> v99.metadata.json, rename it to 00099-uuid.metadata.json
> -- so that BaseMetastoreTableOperations can correctly parse the
> version number.
> 'metadata_location'='s3://existing-table/metadata/00099-uuid.metadata.json',
> 'table_type'='ICEBERG'
> )
> ```
>
> - Others seem to have had success implementing + maintaining a
> custom catalog (https://iceberg.apache.org/custom-catalog/
> <https://iceberg.apache.org/custom-catalog/>) backed by e.g. DynamoDB
> for atomic metadata updates, which could appeal to us. Seems like
> migration in this case consists of implementing the catalog and
> plopping the latest metadata into the backing store. Are custom
> catalogs more of an escape hatch when HMS can’t be used, or would that
> maybe be a reasonable way forward if we find we don’t want to maintain
> + operate on top of HMS?
>
> Apologies if this was discussed or documented somewhere else and I’ve
> missed it.
>
> Thanks!
>
> Marko