You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Chaho12 (via GitHub)" <gi...@apache.org> on 2024/04/08 02:22:12 UTC
[I] Hive metastore does not update metdadata durring commit. [iceberg]
Chaho12 opened a new issue, #10101:
URL: https://github.com/apache/iceberg/issues/10101
### Apache Iceberg version
1.4.3
### Query engine
Hive
### Please describe the bug 🐞
As we all know through the commit process, iceberg registers the most up-to-date metadata file path to the table properties in the hive metastore.
However, in some cases, I notice that even if iceberg registered the path of the metadata file in the hive metastore and received a normal response, but it was `not registered in the hive metastore`, causing the issue.
Since from the perspective of flink iceberg, it is correct to proceed to the next checkpoint because hms responded normally so this unregistered metadata gets lost.
Does anyone have any ideas on fixing this?
```
2024-03-19 03:31:02,535 INFO org.apache.iceberg.flink.sink.IcebergFilesCommitter [] - Start to flush snapshot state to state backend, table: hive.custom_schema.public_hms, checkpointId: 36569
2024-03-19 03:31:02,712 INFO org.apache.hadoop.io.compress.CodecPool [] - Got brand-new compressor [.zstd]
2024-03-19 03:31:03,127 INFO org.apache.iceberg.flink.sink.IcebergFilesCommitter [] - Committing append for checkpoint 36569 to table hive.custom_schema.public_hms branch main with summary: CommitSummary{dataFilesCount=10, dataFilesRecordCount=374, dataFilesByteCount=72923, deleteFilesCount=0, deleteFilesRecordCount=0, deleteFilesByteCount=0}
2024-03-19 03:31:03,506 INFO org.apache.hadoop.io.compress.CodecPool [] - Got brand-new compressor [.zstd]
2024-03-19 03:31:03,966 INFO org.apache.iceberg.hive.HiveTableOperations [] - Committed to table hive.custom_schema.public_hms with the new metadata location hdfs://.../user/user_hive/warehouse/custom_schema.db/public_hms/metadata/36767-0e5f399a-867c-49de-90e1-0806482ea2fa.metadata.json
2024-03-19 03:31:04,042 INFO org.apache.iceberg.BaseMetastoreTableOperations [] - Successfully committed to table hive.custom_schema.public_hms in 611 ms
2024-03-19 03:31:04,042 INFO org.apache.iceberg.SnapshotProducer [] - Committed snapshot 517916272951067890 (MergeAppend)
2024-03-19 03:31:04,196 WARN org.apache.iceberg.SnapshotProducer [] - Failed to load committed snapshot, skipping manifest clean-up
2024-03-19 03:31:04,329 WARN org.apache.iceberg.MergingSnapshotProducer [] - Failed to load committed snapshot: omitting sequence number from notifications
2024-03-19 03:31:04,329 INFO org.apache.iceberg.metrics.LoggingMetricsReporter [] - Received metrics report: CommitReport{tableName=hive.custom_schema.public_hms, snapshotId=s, sequenceNumber=-1, operation=append, commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT1.069239305S, count=1}, attempts=CounterResult{unit=COUNT, value=1}, addedDataFiles=CounterResult{unit=COUNT, value=10}, removedDataFiles=null, totalDataFiles=null, addedDeleteFiles=null, addedEqualityDeleteFiles=null, addedPositionalDeleteFiles=null, removedDeleteFiles=null, removedEqualityDeleteFiles=null, removedPositionalDeleteFiles=null, totalDeleteFiles=null, addedRecords=CounterResult{unit=COUNT, value=374}, removedRecords=null, totalRecords=null, addedFilesSizeInBytes=CounterResult{unit=BYTES, value=72923}, removedFilesSizeInBytes=null, totalFilesSizeInBytes=null, addedPositionalDeletes=null, removedPositionalDeletes=null, totalPositionalDeletes=null, add
edEqualityDeletes=null, removedEqualityDeletes=null, totalEqualityDeletes=null}, metadata={engine-version=1.15.2, engine-name=flink, iceberg-version=Apache Iceberg 1.2.0 (commit e340ad5be04e902398c576f431810c3dfa4fe717)}}
```
<hadoop.version>3.1.0</hadoop.version>
<hive.version>3.1.2</hive.version>
<metastore.version>3.1.3</metastore.version>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hive metastore does not update metdadata durring commit. [iceberg]
Posted by "Chaho12 (via GitHub)" <gi...@apache.org>.
Chaho12 commented on issue #10101:
URL: https://github.com/apache/iceberg/issues/10101#issuecomment-2041913419
@pvary yeah it is similiar issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Hive metastore does not update metdadata durring commit. [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #10101:
URL: https://github.com/apache/iceberg/issues/10101#issuecomment-2041885473
Is this similar to #9753?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org