Re: [PR] [iceberg] support iceberg rest catalog for iceberg compatibility [paimon]

["novakov-alexey (via GitHub)" <gi...@apache.org>]

novakov-alexey commented on PR #5817:
URL: https://github.com/apache/paimon/pull/5817#issuecomment-3691741599

   @LsomeYeah Hi, it seems there is some i encountered on Paimon 1.3.1. 
   
   I have tested main case: 
   - I created Paimon table with REST catalog metadata properties and then wrote some data into it via Flink:
   
   In result, an NPE was thrown on the Flink writer operator. 
   
   ```java
   java.lang.RuntimeException: java.lang.RuntimeException: Fail to create table or get table: default.my_orders
   	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadata(IcebergRestMetadataCommitter.java:123)
   	at org.apache.paimon.iceberg.IcebergCommitCallback.createMetadataWithBase(IcebergCommitCallback.java:666)
   	at org.apache.paimon.iceberg.IcebergCommitCallback.createMetadata(IcebergCommitCallback.java:281)
   	at org.apache.paimon.iceberg.IcebergCommitCallback.call(IcebergCommitCallback.java:229)
   	at org.apache.paimon.operation.FileStoreCommitImpl.lambda$tryCommitOnce$16(FileStoreCommitImpl.java:1215)
   	at java.base/java.util.ArrayList.forEach(Unknown Source)
   	at org.apache.paimon.operation.FileStoreCommitImpl.tryCommitOnce(FileStoreCommitImpl.java:1213)
   	at org.apache.paimon.operation.FileStoreCommitImpl.tryCommit(FileStoreCommitImpl.java:840)
   	at org.apache.paimon.operation.FileStoreCommitImpl.commit(FileStoreCommitImpl.java:362)
   	at org.apache.paimon.table.sink.TableCommitImpl.commitMultiple(TableCommitImpl.java:229)
   	at org.apache.paimon.flink.sink.StoreCommitter.commit(StoreCommitter.java:111)
   	at org.apache.paimon.flink.sink.CommitterOperator.commitUpToCheckpoint(CommitterOperator.java:215)
   	at org.apache.paimon.flink.sink.CommitterOperator.notifyCheckpointComplete(CommitterOperator.java:192)
   	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:104)
   	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.notifyCheckpointComplete(RegularOperatorChain.java:145)
   	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpoint(SubtaskCheckpointCoordinatorImpl.java:479)
   	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:412)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1578)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$20(StreamTask.java:1519)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$23(StreamTask.java:1558)
   	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:118)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:415)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:400)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:362)
   	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:980)
   	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917)
   	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
   	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:942)
   	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
   	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
   	at java.base/java.lang.Thread.run(Unknown Source)
   Caused by: java.lang.RuntimeException: Fail to create table or get table: default.my_orders
   	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadataImpl(IcebergRestMetadataCommitter.java:167)
   	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadata(IcebergRestMetadataCommitter.java:121)
   	... 32 more
   Caused by: java.lang.NullPointerException
   	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.checkBase(IcebergRestMetadataCommitter.java:355)
   	at org.apache.paimon.iceberg.IcebergRestMetadataCommitter.commitMetadataImpl(IcebergRestMetadataCommitter.java:154)
   	... 33 more
   ```
   
   It seems code logic thinks - there is some `currentSnapshot` with snapshotId. All what I see that Flink has created is a metadata file in JSON format in the REST Catalog warehouse S3 folder. 
   One of the field inside the metadata file on is: 
   ```json
   "current-snapshot-id":-1
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org