You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "chenwyi2 (via GitHub)" <gi...@apache.org> on 2023/11/15 03:34:41 UTC
[I] when hdfs router restart, task failed with read data files " xx parquet is not a parquet file (length is too low :0) " or manifest "java.io.EOFException: Unexpected EOF with 4 bytes remaining to read avro" [iceberg]
chenwyi2 opened a new issue, #9071:
URL: https://github.com/apache/iceberg/issues/9071
### Apache Iceberg version
1.2.1
### Query engine
None
### Please describe the bug 🐞
recently, we found that when hdfs router has stuck or restart, we will meet some files are empty, for example, we have a flink task write iceberg every one minute, router has restart during writing and client has java.net.SocketTimeoutException or "A failover has occurred since the start of call create" such exception, but task doesn't failed, then i read this commit with error " xx parquet is not a parquet file (length is too low :0) ".
i list the file and the length is zero, and i look hdfs audit log, i find this file was created two times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] when hdfs router restart, task failed with read data files " xx parquet is not a parquet file (length is too low :0) " or manifest "java.io.EOFException: Unexpected EOF with 4 bytes remaining to read avro" [iceberg]
Posted by "link3280 (via GitHub)" <gi...@apache.org>.
link3280 commented on issue #9071:
URL: https://github.com/apache/iceberg/issues/9071#issuecomment-1925554243
We get the same problem here with Iceberg 1.3.0.
The bug affects not only data files but also metadata.json and .avro files. The files created twice could be corrupted (1-2% chance in my case).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] when hdfs router restart, task failed with read data files " xx parquet is not a parquet file (length is too low :0) " or manifest "java.io.EOFException: Unexpected EOF with 4 bytes remaining to read avro" [iceberg]
Posted by "chenwyi2 (via GitHub)" <gi...@apache.org>.
chenwyi2 closed issue #9071: when hdfs router restart, task failed with read data files " xx parquet is not a parquet file (length is too low :0) " or manifest "java.io.EOFException: Unexpected EOF with 4 bytes remaining to read avro"
URL: https://github.com/apache/iceberg/issues/9071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org