You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/06/16 08:48:51 UTC
[GitHub] [hudi] LinMingQiang opened a new issue, #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
LinMingQiang opened a new issue, #5886:
URL: https://github.com/apache/hudi/issues/5886
**When debugging, it is found that the next commit will throw an exception after the clean is completed**
I found that `HoodieTableFileSystemView.partitionToFileGroupsMap` lost the last `instant commit` fileGoup infomation.
(I execute `hoodieTable.getHoodieView().reset()` after Throw Exception，and its working after retry `getLatestBaseFile`)
hudi table config：
```
"'table.type' = 'COPY_ON_WRITE',\n" +
"'hoodie.parquet.small.file.limit' = '20', \n" +
"'write.operation' = 'insert', \n" +
"'write.insert.cluster' = 'true', \n" +
"'hoodie.datasource.write.hive_style_partitioning' = 'true',\n" +
"'write.task.max.size' = '4096', \n" +
"'write.merge.max_memory'= '2048',\n" +
"'write.precombine' = 'true',\n" +
"'write.tasks' = '1',\n" +
"'write.bucket_assign.tasks' = '1',\n" +
"'hive_sync.skip_ro_suffix' = 'true',\n" +
"'write.ignore.failed' = 'true',\n" +
"'clean.async.enabled' = 'true',\n" +
"'clean.retain_commits' = '6' \n" +
```
```
The determining factor is
'hoodie.parquet.small.file.limit' = '20' -- Trigger new file generation
and
'clean.async.enabled' = 'true' -- Trigger async clean
'clean.retain_commits' = '6'
```
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1157647695
Can you share the full stacktrace, and timeline under `.hoodie` dir?
Btw, `'hoodie.parquet.small.file.limit' = '20'` is way too small. This config takes file size limit in bytes. Any particular reason for such a small value? Did you try out with default configs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1159373599
> Sorry, I am not following completely. The file system view during inflight commit (i.e. `20220617091358406.inflight`) after cleaning (i.e `20220617091348754.clean`) should have 3 instants: `20220617091324896.commit, 20220617091336735.commit, 20220617091341327.commit`? Are you suggesting that you see only the first two commits and not the third one? Ideally this should not happen because for each commit file system view is instantiated with visible commits. Is it possible due to race that cleaning completed before `20220617091341327.commit`/
Yes, as you understand. When I debug in `AbstractTableFileSystemView.addFilesToView` , At moments `20220617091341327` and `20220617091358406 ` The value of parameter
`visibleCommitsAndCompactionTimeline` has only the first two but no third.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1163643955
@LinMingQiang : So, with the linked patch, I assume things are good? can we close the github issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang closed issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang closed issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
URL: https://github.com/apache/hudi/issues/5886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1159397982
And clean completed always finishes before xx38725.commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1158504491
This problem is caused by the fact that the timeline is not synchronized after the clean is completed. Cause `AbstractTableFileSystemView.visibleCommitsAndCompactionTimeline` s lastInstant to stay at the first two times.
This causes the latest commit fileslice to be filtered out in `HoodieFileGroup.isFileSliceCommitted`. So I think we should take the initiative to do sync once when clean is completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1159378452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1159397919
> 
And `clean completed` always finishes before `xx38725.commit`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1159397634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1158381755
This problem requires a large amount of data to be written, 'hoodie.parquet.small.file.limit' = '20' can be a quick recurrence of the problem.. and I found that after 'clean' is completed, this error will appear in the next instant commit, And the nonexistent fileID must be the last instant commit.
During debugging, I found that:
```
```
1：The error occurs in the first commit after clean
```
-rw-r--r-- 1 hunter staff 5646 Jun 17 09:13 20220617091324896.commit
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091324896.commit.requested
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091324896.inflight
-rw-r--r-- 1 hunter staff 4028 Jun 17 09:13 20220617091336735.commit
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091336735.commit.requested
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091336735.inflight
-rw-r--r-- 1 hunter staff 5650 Jun 17 09:13 20220617091341327.commit
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091341327.commit.requested
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091341327.inflight
-rw-r--r-- 1 hunter staff 1561 Jun 17 09:13 20220617091348754.clean
-rw-r--r-- 1 hunter staff 1699 Jun 17 09:13 20220617091348754.clean.inflight
-rw-r--r-- 1 hunter staff 1699 Jun 17 09:13 20220617091348754.clean.requested
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091358406.commit.requested
-rw-r--r-- 1 hunter staff 0 Jun 17 09:13 20220617091358406.inflight
```
```
-rw-r--r-- 1 hunter staff 7772443 Jun 17 09:12 536b7835-0754-4187-96dc-1a5ecc5c152a_0-1-0_20220617091245620.parquet
-rw-r--r-- 1 hunter staff 12079980 Jun 17 09:13 536b7835-0754-4187-96dc-1a5ecc5c152a_0-1-0_20220617091257227.parquet
-rw-r--r-- 1 hunter staff 12125690 Jun 17 09:13 536b7835-0754-4187-96dc-1a5ecc5c152a_0-1-0_20220617091302805.parquet
-rw-r--r-- 1 hunter staff 5261158 Jun 17 09:13 5ab5fc26-12bc-4832-b70c-499ef968b427_0-1-0_20220617091324896.parquet
-rw-r--r-- 1 hunter staff 8276875 Jun 17 09:13 5ab5fc26-12bc-4832-b70c-499ef968b427_0-1-0_20220617091336735.parquet
-rw-r--r-- 1 hunter staff 8322925 Jun 17 09:13 5ab5fc26-12bc-4832-b70c-499ef968b427_0-1-0_20220617091341327.parquet
-rw-r--r-- 1 hunter staff 18307016 Jun 17 09:12 61b5598d-7879-439b-87d8-04ce14b95a4e_0-1-0_20220617091237415.parquet
-rw-r--r-- 1 hunter staff 18356323 Jun 17 09:12 61b5598d-7879-439b-87d8-04ce14b95a4e_0-1-0_20220617091245620.parquet
-rw-r--r-- 1 hunter staff 18646634 Jun 17 09:13 b4c27901-1c35-4224-8485-8540278d85c3_0-1-0_20220617091341327.parquet
-rw-r--r-- 1 hunter staff 14465106 Jun 17 09:13 d9b1e244-098c-487b-b927-4c76bb6a8e97_0-1-0_20220617091302805.parquet
-rw-r--r-- 1 hunter staff 16797868 Jun 17 09:13 d9b1e244-098c-487b-b927-4c76bb6a8e97_0-1-0_20220617091317899.parquet
-rw-r--r-- 1 hunter staff 16844180 Jun 17 09:13 d9b1e244-098c-487b-b927-4c76bb6a8e97_0-1-0_20220617091324896.parquet
```
2： `HoodieTableFileSystemView.partitionToFileGroupsMap` sometimes not contains FileGoups ``，but sometimes do.
so when i execute `hoodieTable.getMetadataTable().reset()` after `HoodieMergeHandler.getLatestBaseFile` throw error and re-execute `hoodieTable.getBaseFileOnlyView().getLatestBaseFile(partitionPath, fileId)` it working well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1158889882
Sorry, I am not following completely.
The file system view during inflight commit (i.e. `20220617091358406.inflight`) after cleaning (i.e `20220617091348754.clean`) should have 3 instants: `20220617091324896.commit, 20220617091336735.commit, 20220617091341327.commit`? Are you suggesting that you see only the first two commits and not the third one?
Ideally this should not happen because for each commit file system view is instantiated with visible commits. Is it possible due to race that cleaning completed before `20220617091341327.commit`/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on issue #5886: Throw `NoSuchElementException: FileID xx of partition path xx does not exist.` when execute `HoodieMergeHandle.getLatestBaseFile` but FileID is exist in path.
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on issue #5886:
URL: https://github.com/apache/hudi/issues/5886#issuecomment-1159407490
The modified method can actively perform sync at the time of `instant commit`, or remove `clean instant` when judging `islocalviewbehind`, and then judge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org