Metadata error handling

[Peter Vary <pv...@cloudera.com.INVALID>]

Hi Team,

I have been playing around with Iceberg tables, and I had some failing writes (not really sure how it actually happened), and ended up with an empty metadata/version-hint.text file.
When I have tried to read this table with Catalogs.loadTable(configuration, properties) I got a NullPointerException.

How do we handle metadata corruption in Iceberg?
Shall we try to fix this error? - Maybe based on the other metadata files we can find the actual version
Provide a better error message?
Or just make sure a developer can find out what is the problem, so the NullPointer exception is ok?

Thanks,
Peter

Re: Metadata error handling

[Ryan Blue <rb...@netflix.com.INVALID>]

A missing or corrupt version hint file shouldn't cause a failure. If the
hint is not available, then Iceberg should find the latest metadata version
by checking for the existence of metadata files. I think we just need to
add a warning and ignore this error. Can you open a PR?

rb

On Thu, Sep 10, 2020 at 6:46 AM Peter Vary <pv...@cloudera.com.invalid>
wrote:

> Hi Team,
>
> I have been playing around with Iceberg tables, and I had some failing
> writes (not really sure how it actually happened), and ended up with an
> empty metadata/version-hint.text file.
> When I have tried to read this table with Catalogs.loadTable(configuration
> , properties) I got a NullPointerException.
>
> How do we handle metadata corruption in Iceberg?
>
>    1. Shall we try to fix this error? - Maybe based on the other metadata
>    files we can find the actual version
>    2. Provide a better error message?
>    3. Or just make sure a developer can find out what is the problem, so
>    the NullPointer exception is ok?
>
>
> Thanks,
> Peter
>


-- 
Ryan Blue
Software Engineer
Netflix