You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "BsoBird (via GitHub)" <gi...@apache.org> on 2023/12/18 03:21:07 UTC
[I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
BsoBird opened a new issue, #9327:
URL: https://github.com/apache/iceberg/issues/9327
### Apache Iceberg version
1.4.2 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
SPARK 3.4.1.
```
Caused by: java.io.FileNotFoundException: File does not exist: /iceberg-catalog/warehouse/dwd/b_std_category/data/00012-4526569-b66acfb2-bea0-46af-a6c8-01d9d1731b35-00001.orc
```
我们发现,在一些情况下,ICEBERG表可能会出现FileNotFoundException.
事情的经过是这样子的:
1.我们每天凌晨1点会开始执行8个表的MERGE INTO操作. 在操作完成后,会执行如下三个操作:
```
CALL xxx.system.rewrite_manifests('dwd.b_std_category', false);
CALL xxx.system.remove_orphan_files(table => 'dwd.b_std_category');
CALL xxx.system.expire_snapshots(table => 'dwd.b_std_category', retain_last => 10);
```
2.今天执行MERGE操作时,发生了OOM.导致容器被杀死
```
Job aborted due to stage failure: Authorized committer (attemptNumber=0, stage=415011, partition=1392) failed; but task commit success, data duplication may happen. reason=ExecutorLostFailure(3759,true,Some(Container killed by YARN for exceeding physical memory limits. 40.1 GB of 40 GB physical memory used. Consider boosting spark.executor.memoryOverhead.))
```
3.发生OOM时,dwd.b_std_category 表正在执行这一句指令:
```
CALL xxx.system.expire_snapshots(table => 'dwd.b_std_category', retain_last => 10);
```
4.当我们重新恢复SPARK任务后,我们发现dwd.b_std_category表无法被读取.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859523079
@RussellSpitzer
--User has mutliple tables homed in the same directory, Remove orphan files for one deletes files for the other. This is generally unrecoverable and you'll notice that many files do not exist that should according to iceberg metadata
no,Our catalogues and tables are one-to-one.
--User error, manually deleting a file.
no. No one but me can manipulate iceberg's data files. And I just run the CALL command every day.
--Third party ttl system
no. we haven't.
--The query is running a snapshot which has been expired but not yet removed from cache, or the query was started and then the snapshot was expired. In this case you just refresh the cache and the query will work
How do I flush the cache? I've restarted the spark job. I've restarted the spark job. If it's an in-memory cache, then it shouldn't have a problem. If it's cached in memory, then it shouldn't be a problem, and I do have a snapshot of the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859711825
I'm here to submit a PR for a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859515258
When I see this sort of thing it's usually one of two issues
1. The user has accidentally run some command which deletes files without talking to iceberg, the snapshot refers to a file which no longer exists. This was done accidentally so the file must be removed via a metadata delete to get the table back to a healthy status.
Common reasons for this
* User has mutliple tables homed in the same directory, Remove orphan files for one deletes files for the other
* User error, manually deleting a file
* Third party ttl system
2. The query is running a snapshot which has been expired but not yet removed from cache, or the query was started and then the snapshot was expired. In this case you just refresh the cache and the query will work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1867371609
@Zhangg7723 Our data table does not have concurrent operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859496024
@chennurchaitanya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859495578
I also found another situation where I have a table that has never had a CALL command executed on it. All I did was to run a MERGE INTO once a day, but after the OOM, the table did happend same thing. It's been so long that I didn't keep a log of that scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]
Posted by "Zhangg7723 (via GitHub)" <gi...@apache.org>.
Zhangg7723 commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1867393450
step 3:At the time of the OOM, the dwd.b_std_category table was executing this command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1867408622
Does that cause any problems?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859497363
Slack link: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1700291064787019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859492897
@RussellSpitzer @nastra
can you help me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859649309
@RussellSpitzer
I have slightly modified HadoopTableOperations.commit(). What do you think about this modification?
```
@Override
public void commit(TableMetadata base, TableMetadata metadata) {
Pair<Integer, TableMetadata> current = versionAndMetadata();
if (base != current.second()) {
throw new CommitFailedException("Cannot commit changes based on stale table metadata");
}
if (base == metadata) {
LOG.info("Nothing to commit.");
return;
}
Preconditions.checkArgument(
base == null || base.location().equals(metadata.location()),
"Hadoop path-based tables cannot be relocated");
Preconditions.checkArgument(
!metadata.properties().containsKey(TableProperties.WRITE_METADATA_LOCATION),
"Hadoop path-based tables cannot relocate metadata");
int nowVersion = (current.first() != null ? current.first() : 0);
int nextVersion = nowVersion + 1;
boolean versionCommit = false;
try{
String codecName =
metadata.property(
TableProperties.METADATA_COMPRESSION, TableProperties.METADATA_COMPRESSION_DEFAULT);
TableMetadataParser.Codec codec = TableMetadataParser.Codec.fromName(codecName);
String fileExtension = TableMetadataParser.getFileExtension(codec);
Path tempMetadataFile = metadataPath(UUID.randomUUID().toString() + fileExtension);
TableMetadataParser.write(metadata, io().newOutputFile(tempMetadataFile.toString()));
Path finalMetadataFile = metadataFilePath(nextVersion, codec);
FileSystem fs = getFileSystem(tempMetadataFile, conf);
// this rename operation is the atomic commit operation
renameToFinal(fs, tempMetadataFile, finalMetadataFile, nextVersion);
LOG.info("Committed a new metadata file {}", finalMetadataFile);
// update the best-effort version pointer
writeVersionHint(nextVersion);
versionCommit = true;
//perhaps this step will throw exception
deleteRemovedMetadataFiles(base, metadata);
this.shouldRefresh = true;
}catch (Exception e){
if(versionCommit){
//try best roll back,if fail,user should roll back version by hem self
writeVersionHint(nowVersion);
}
throw e;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]
Posted by "Zhangg7723 (via GitHub)" <gi...@apache.org>.
Zhangg7723 commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1867366284
Was the MERGE operation committed successfully? if it was successfully and committed files were removed by remove_orphan_files job, the latest snapshot will refer to lost files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-2594192645
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG) [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9327: FileNotFoundException can occur in some scenarios. (data file & HADOOP CATALOG)
URL: https://github.com/apache/iceberg/issues/9327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859711384
I have identified the problem, during the execution of deleteRemovedMetadataFiles(base, metadata), if there is some unexpected problem such as OOM, it will cause the datafile to be cleaned up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org