You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/10/24 11:38:43 UTC
[GitHub] [ozone] Xushaohong opened a new pull request, #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Xushaohong opened a new pull request, #3876:
URL: https://github.com/apache/ozone/pull/3876
## What changes were proposed in this pull request?
1. add detection of unreferenced chunks in full data scan
2. add contanierGarbageCollector in **KeyValueHandler** to collect and remove the orphan files
3. move the function **getChunkDir** from **ContainerLayoutVersion** to **ContainerUtils**
This aims to solve the case B in
https://docs.google.com/document/d/1Oi1zPKdmvA7DFwIcUWz6zw_p-pY3D1L76gyV37jQT94/edit#
Currently, there is no report of such orphan chunk info forward to Recon/SCM. We directly delete them serially after a full data scan with a few checks to avoid possible accidents. Maybe we could add a recon to imply this info and add a button to trigger the removal.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-1202
## How was this patch tested?
1. Created a POC test and tested deleting the orphan blocks created by EC write.
2. UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3876:
URL: https://github.com/apache/ozone/pull/3876#issuecomment-1289278037
cc @dombizita
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] xBis7 commented on pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on PR #3876:
URL: https://github.com/apache/ozone/pull/3876#issuecomment-1409048627
@Xushaohong @kerneltime Since no new test cases for orphan chunk detection have been brought up, should this be closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Xushaohong commented on pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by GitBox <gi...@apache.org>.
Xushaohong commented on PR #3876:
URL: https://github.com/apache/ozone/pull/3876#issuecomment-1290292858
The original design of https://issues.apache.org/jira/browse/HDDS-1084 implies the following:
The Recon UI as a sub component of Ozone Console provides the following:
- Data browser for Keys -> Containers, Keys -> missing blocks
- Reporting use cases: Orphaned blocks, Corrupt blocks :- grouped by datanode
- Query editor: Allow user to query the aggregate SQL DB for analytics
To satisfy the future usage of recon for Orphaned blocks, I will update more commits to construct the API for the report of Orphaned blocks to Recon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Xushaohong commented on pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by GitBox <gi...@apache.org>.
Xushaohong commented on PR #3876:
URL: https://github.com/apache/ozone/pull/3876#issuecomment-1298254805
According to the discussion on 10.28, this completely guaranteed orphan chunk detection is not recommended unless there are extra specific scenarios relying on this.
Only the allocated blocks should be cleaned up and it could be solved by the combination of Case A CaseB CaseC, and their respective solutions except this(The details are in Jira).
I would follow up and find out possible cases that need this during the test.
If anyone could point out such cases and clarify the necessity, that would be a great help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Xushaohong closed pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by "Xushaohong (via GitHub)" <gi...@apache.org>.
Xushaohong closed pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
URL: https://github.com/apache/ozone/pull/3876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on PR #3876:
URL: https://github.com/apache/ozone/pull/3876#issuecomment-1292663795
> The original design of https://issues.apache.org/jira/browse/HDDS-1084 implies the following:
>
> The Recon UI as a sub component of Ozone Console provides the following:
>
> * Data browser for Keys -> Containers, Keys -> missing blocks
> * Reporting use cases: Orphaned blocks, Corrupt blocks :- grouped by datanode
> * Query editor: Allow user to query the aggregate SQL DB for analytics
>
> To satisfy the future usage of recon for Orphaned blocks, I will update more commits to construct the API for the report of Orphaned blocks to Recon.
Thanks Hong for the context! (reviewing!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by GitBox <gi...@apache.org>.
kerneltime commented on PR #3876:
URL: https://github.com/apache/ozone/pull/3876#issuecomment-1289276880
cc @aswinshakil @DaveTeng0 @duongkame
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Xushaohong commented on pull request #3876: HDDS-1202. Detection and Removal of Orphan Chunks at the Container Level
Posted by "Xushaohong (via GitHub)" <gi...@apache.org>.
Xushaohong commented on PR #3876:
URL: https://github.com/apache/ozone/pull/3876#issuecomment-1409631812
This fix is covered by https://issues.apache.org/jira/browse/HDDS-7445 and hence closes the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org