[DISCUSS] Table updates in a REST catalog

[Ryan Blue <bl...@tabular.io>]

Hi everyone,

I wrote up a design doc that covers making changes to Iceberg tables in the
REST catalog API
<https://docs.google.com/document/d/1D0R3G0slssEhggH5XnIzMwsUIP-c385Qp2sjv5E7e6E/edit?usp=sharing>
.

The doc outlines the current changes to metadata, identifies a more
orthogonal set of changes that we can use, and outlines conflict detection.
In the end, the proposed API is to send a list of requirements, like
"current snapshot must be <ID>", and a list of changes, like "add snapshot
..." and "set current snapshot to <ID>".

Please have a look and discuss!

Ryan

-- 
Ryan Blue
Tabular

Re: [DISCUSS] Table updates in a REST catalog

[Ryan Blue <bl...@tabular.io>]

Hi Yufei, I’ll reply to your questions inline:

On Tue, Jan 4, 2022 at 3:12 PM Yufei Gu <fl...@gmail.com> wrote:

1. Are we going to open for other ways to store the table metadata instead
> of the metadata.json files? For example, a relational database or a
> key-value database. This will be a big change if that’s so, since it
> changes the assumption that the table version is preserved as a file.
>
We can relax the spec here in the future. My original intent with the
TableOperations API was to allow flexibility to use a database, but the
metadata JSON files have been an important part of compatibility. For now,
this is required by the spec, but there is nothing stopping a catalog
implementation from not writing the metadata JSON file.

2. Assume that we still keep metadata.json files, does that mean the
> catalog server needs to write a new metadata.json for each commit, which
> needs the permission to access the file system where the table is located?
> It is tricky to do that for some users whose catalog doesn’t have the
> access of the file system.
>
That’s right. To follow the spec, you’d need the service to write a
metadata JSON file, although I don’t think it is likely that the client
would actually attempt to read it — as long as the service returns
metadata, it would just be extra work. So you could choose whether to make
sure your service can write the file, or privately violate the spec. My
recommendation here is to make sure the service can write the file. It
provides an important backup of table state in case anything happens to the
DB, and is the most portable representation of table metadata.

3. What’s the major benefits of the REST catalog APIs? Considering that the
> current user has to make a lot of changes to adopt it, either from client
> side and server side. The client side should be fine since Iceberg may
> provide it, but the server side is a big task for existing users.
>
There are quite a few benefits:

   - It is a standard client interface that can be used by processing
   engines. It’s hard to get a catalog client into products like Trino.
   Organizations that use a custom metastore can’t expect to get a Jar into
   hosted processing services, so it makes sense to use one common client and
   customize a service endpoint.
   - Customizing the service with a standard client also allows writing
   just one catalog implementation that can be used across languages, rather
   than trying to implement the catalog the same way across Java, Python, etc.
   - A catalog service can do a better job with resources like JDBC
   connections. You probably don’t want hundreds or thousands of connections
   directly to the catalog database.
   - A change-based API has several advantages as well:
      - Conflict detection can be more granular: commits to different
      branches don’t need to conflict with one another, nor should metadata
      commits
      - Only one writer version is producing metadata JSON files so you
      don’t have to update every writer in your infrastructure to use new
      metadata structures
      - Reads and writes send data directly to/from S3 or another object
      store because metadata JSON writes are handled by the service

4. How does an existing catalog support the new APIs? For example, HMS may
> need to add an extra layer or a plugin to support the server side of the
> APIs.
>
The existing catalogs will continue to work as they are. There’s no need
for HMS to change since it already has a running service and working API.
But this would benefit other catalogs like JDBC (as I noted above) and
would help with cross-language compatibility. I think the advantages are
significant enough that we will probably see most people choosing to use
the REST catalog, but I could be wrong.

Ryan
-- 
Ryan Blue
Tabular

Re: [DISCUSS] Table updates in a REST catalog

[Yufei Gu <fl...@gmail.com>]

Hi Ryan,

Thanks for the proposal. Here are some questions and comments.

   1. Are we going to open for other ways to store the table metadata
   instead of the metadata.json files? For example, a relational database or a
   key-value database. This will be a big change if that’s so, since it
   changes the assumption that the table version is preserved as a file.
   2. Assume that we still keep metadata.json files, does that mean the
   catalog server needs to write a new metadata.json for each commit, which
   needs the permission to access the file system where the table is located?
   It is tricky to do that for some users whose catalog doesn’t have the
   access of the file system.
   3. What’s the major benefits of the REST catalog APIs? Considering that
   the current user has to make a lot of changes to adopt it, either from
   client side and server side. The client side should be fine since Iceberg
   may provide it, but the server side is a big task for existing users.
   4. How does an existing catalog support the new APIs? For example, HMS
   may need to add an extra layer or a plugin to support the server side of
   the APIs.

Best,

Yufei

`This is not a contribution`


On Mon, Jan 3, 2022 at 10:42 AM Ryan Blue <bl...@tabular.io> wrote:

> Hi everyone,
>
> I wrote up a design doc that covers making changes to Iceberg tables in
> the REST catalog API
> <https://docs.google.com/document/d/1D0R3G0slssEhggH5XnIzMwsUIP-c385Qp2sjv5E7e6E/edit?usp=sharing>
> .
>
> The doc outlines the current changes to metadata, identifies a more
> orthogonal set of changes that we can use, and outlines conflict detection.
> In the end, the proposed API is to send a list of requirements, like
> "current snapshot must be <ID>", and a list of changes, like "add snapshot
> ..." and "set current snapshot to <ID>".
>
> Please have a look and discuss!
>
> Ryan
>
> --
> Ryan Blue
> Tabular
>