You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "cgpoh (via GitHub)" <gi...@apache.org> on 2023/03/18 07:01:36 UTC
[GitHub] [iceberg] cgpoh opened a new issue, #7137: Avro referenced in metadata json file is missing
cgpoh opened a new issue, #7137:
URL: https://github.com/apache/iceberg/issues/7137
### Apache Iceberg version
1.1.0 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
We are having this issue https://github.com/apache/iceberg/issues/4472#issue-1190603304 of missing Avro file reference in the metadata json. This cause S3 keynotfound exception and unable to start our Flink job. The workaround we did is manually remove the missing Avro file in the metadata json file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Avro referenced in metadata json file is missing [iceberg]
Posted by "cgpoh (via GitHub)" <gi...@apache.org>.
cgpoh commented on issue #7137:
URL: https://github.com/apache/iceberg/issues/7137#issuecomment-1744605057
I'm facing this issue again today. The manifest-list is pointing to a non-existent file and it is throwing `The specified key does not exist. (Service: S3, Status Code: 404, Request ID: 178A8EA8049ED525)` error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Avro referenced in metadata json file is missing [iceberg]
Posted by "cgpoh (via GitHub)" <gi...@apache.org>.
cgpoh commented on issue #7137:
URL: https://github.com/apache/iceberg/issues/7137#issuecomment-2167500208
I'm facing this issue again today. I think the cause of the problem is our Hive Metastore ran out of memory and before it crashes, it created the metadata file and cause the key not found issue. To resolve this issue, I have to manually edit the metadata json file to force it to use the previous snapshot id instead of the corrupted id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] marvin-roesch commented on issue #7137: Avro referenced in metadata json file is missing
Posted by "marvin-roesch (via GitHub)" <gi...@apache.org>.
marvin-roesch commented on issue #7137:
URL: https://github.com/apache/iceberg/issues/7137#issuecomment-1683725234
I can confirm that we are experiencing the same issue when inserting records into an Iceberg table using Trino. A commit might fail and will leave the metadata for the table in a bad state, resulting in any subsequent actions to fail as well.
Using Trino version 422.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Avro referenced in metadata json file is missing [iceberg]
Posted by "erikcw (via GitHub)" <gi...@apache.org>.
erikcw commented on issue #7137:
URL: https://github.com/apache/iceberg/issues/7137#issuecomment-1775505118
I'm also experiencing this issue. I'm on Trino version 430.
```
çorg.apache.iceberg.exceptions.RuntimeIOException: Failed to open file: s3://MY.BUCKET/parquet/campaigns-pre-id-iceberg/metadata/5e73263d-42dd-4e6a-af4b-3027ec72896a-m25.avro
at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:102)
at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:76)
at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:188)
at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:187)
at org.apache.iceberg.io.CloseableIterable.lambda$filter$1(CloseableIterable.java:136)
at org.apache.iceberg.io.CloseableIterable$2.iterator(CloseableIterable.java:72)
at org.apache.iceberg.io.CloseableIterable.lambda$filter$1(CloseableIterable.java:136)
at org.apache.iceberg.io.CloseableIterable$2.iterator(CloseableIterable.java:72)
at org.apache.iceberg.io.CloseableIterable.lambda$filter$1(CloseableIterable.java:136)
at org.apache.iceberg.io.CloseableIterable$2.iterator(CloseableIterable.java:72)
at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:188)
at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:187)
at org.apache.iceberg.ManifestGroup$1.iterator(ManifestGroup.java:333)
at org.apache.iceberg.ManifestGroup$1.iterator(ManifestGroup.java:291)
at org.apache.iceberg.util.ParallelIterable$ParallelIterator.lambda$new$1(ParallelIterable.java:69)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.FileNotFoundException: File s3://MY.BUCKET/parquet/campaigns-pre-id-iceberg/metadata/5e73263d-42dd-4e6a-af4b-3027ec72896a-m25.avro not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 3QPXD22DZE38NQK8; S3 Extended Request ID: i/+9xgHx+b4VsxsUDx2qz+qcm0dc8rDCxGObBEkliJxl7N3tV4Zq3MvHq7uWnVA5iqUdBBuN3mY=; Proxy: null) (Bucket: MY.BUCKET, Key: parquet/campaigns-pre-id-iceberg/metadata/5e73263d-42dd-4e6a-af4b-3027ec72896a-m25.avro)
at io.trino.filesystem.hdfs.HdfsTrinoInputStream.read(HdfsTrinoInputStream.java:103)
at io.trino.plugin.iceberg.fileio.ForwardingSeekableInputStream.read(ForwardingSeekableInputStream.java:66)
at org.apache.iceberg.avro.AvroIO$AvroInputStreamAdapter.read(AvroIO.java:117)
at org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:65)
at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:100)
... 19 more
Caused by: java.io.FileNotFoundException: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 3QPXD22DZE38NQK8; S3 Extended Request ID: i/+9xgHx+b4VsxsUDx2qz+qcm0dc8rDCxGObBEkliJxl7N3tV4Zq3MvHq7uWnVA5iqUdBBuN3mY=; Proxy: null) (Bucket: MY.BUCKET, Key: parquet/campaigns-pre-id-iceberg/metadata/5e73263d-42dd-4e6a-af4b-3027ec72896a-m25.avro)
at io.trino.hdfs.s3.TrinoS3FileSystem.throwIfFileNotFound(TrinoS3FileSystem.java:2074)
at io.trino.hdfs.s3.TrinoS3FileSystem$TrinoS3InputStream.lambda$openStream$2(TrinoS3FileSystem.java:1580)
at io.trino.hdfs.s3.RetryDriver.run(RetryDriver.java:125)
at io.trino.hdfs.s3.TrinoS3FileSystem$TrinoS3InputStream.openStream(TrinoS3FileSystem.java:1557)
at io.trino.hdfs.s3.TrinoS3FileSystem$TrinoS3InputStream.openStream(TrinoS3FileSystem.java:1542)
at io.trino.hdfs.s3.TrinoS3FileSystem$TrinoS3InputStream.seekStream(TrinoS3FileSystem.java:1535)
at io.trino.hdfs.s3.TrinoS3FileSystem$TrinoS3InputStream.lambda$read$1(TrinoS3FileSystem.java:1479)
at io.trino.hdfs.s3.RetryDriver.run(RetryDriver.java:125)
at io.trino.hdfs.s3.TrinoS3FileSystem$TrinoS3InputStream.read(TrinoS3FileSystem.java:1478)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
at java.base/java.io.DataInputStream.read(DataInputStream.java:151)
at java.base/java.io.DataInputStream.read(DataInputStream.java:151)
at io.trino.filesystem.hdfs.HdfsTrinoInputStream.read(HdfsTrinoInputStream.java:100)
... 23 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 3QPXD22DZE38NQK8; S3 Extended Request ID: i/+9xgHx+b4VsxsUDx2qz+qcm0dc8rDCxGObBEkliJxl7N3tV4Zq3MvHq7uWnVA5iqUdBBuN3mY=; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5520)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5467)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1554)
at io.trino.hdfs.s3.TrinoS3FileSystem$TrinoS3InputStream.lambda$openStream$2(TrinoS3FileSystem.java:1563)
... 36 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Avro referenced in metadata json file is missing [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7137:
URL: https://github.com/apache/iceberg/issues/7137#issuecomment-2625997753
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Avro referenced in metadata json file is missing [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7137:
URL: https://github.com/apache/iceberg/issues/7137#issuecomment-2594191683
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Avro referenced in metadata json file is missing [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7137: Avro referenced in metadata json file is missing
URL: https://github.com/apache/iceberg/issues/7137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org