You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "cshannon (via GitHub)" <gi...@apache.org> on 2023/09/18 12:03:39 UTC
[GitHub] [accumulo] cshannon opened a new pull request, #3758: No chop merge and delete
cshannon opened a new pull request, #3758:
URL: https://github.com/apache/accumulo/pull/3758
This PR is to merge in the no-chop merge and no-chop/no-split feature into main. There are still some follow on tasks to do which I will list and create issues for but the main functionality is complete and now that 3.0 has been release I think it's time to get what is here reviewed and merged into main so we can continue on.
The goal here is to prevent having to run chop compactions when merging (or deleting) tablets which can be quite slow. The other goal is to get rid of having to split tablets during deletions to simplify the manager code. Instead, the changes here allow RFiles to be fenced off by by valid ranges so that we don't need to compact/split anymore and when scanning only the valid range is visible. Files can then be cleaned up later with the normal compaction process.
I plan to create a wiki page or website post at some point with full details of how no-chop- merge works and the changes (as it would be quite large) but the highlights are below for reviewing the Pr:
**TabletFile**
Tablet files used to be unique by just the path. Now a TabletFile also includes a Range. This allows Accumulo to track the same file more than one time, each with a different associated range. So the same file can have several valid ranges and be treated as different files. The benefit here is most of the TabletFile handling code can stay the same. If a Range is not provided (null start/end) then the entire file is visible and this is how backwards compability is handled.
**RFiles**
There is now a new RFile iterator/reader that can fence files by taking a collection of ranges and only returning rows that are part of the given range when scanning. Rows outside of the range are skipped and not visible.
**Metadata**
The File metadata that is stored has been changed to now include a range. Originally each file just contained the path in the Data File column. Now this has been changed so that the value is Json and includes both a file and a range. This means that the same file can be stored more than one time in metadata as the combination of file and range makes it unique.
**Merge Strategy**
During a merge, instead of compacting tablets, each tablet that is being merged will instead have its files fenced by the tablet range. This is a quick operation as this just requires going through the files in metadata and adding the range to the entry and then adding that entry to the target tablet where files are being merged to . If an existing range already exists for the file when processing then that file has its range clipped with the tablet range to get the valid overlapping range. Each file will only have 1 range created.
**Delete Strategy**
Deletes are a little more complicated because besides chop compactions when there are shared boundaries (tablets are stretched to cover the deleted range so we need to compact), we also need to handle splits. Normally splits are first done at the deletion boundaries so entire tablets can be deleted but now we can also fence off tablets instead of splitting. Deletion now works by figuring out the first/last tablets in the deletion range and instead of splitting them or compacting the algorithm will fence the tablets (if needed) and create 0, 1 or 2 ranges. Then any tablets that exist in the middle are still just deleted like before.
When the deletion range overlaps part of the first tablet or part of the last tablet we need to create 1 valid range for the files in those tablets (the part of the file not being deleted). This is done instead of a split. If a deletion range is entirely inside of a single tablet then we need to create two ranges (two files are added for each existing file) as we need to fence the file by the valid range before and after the delete range.
**Future Issues to be Done**
1. All of the existing chop compaction related code can be removed (or deprecated if public API). There's a lot to remove and would make the PR much larger so I figured it was best to do a follow on.
2. We still need some more testing to make sure all edge cases are handled
3. Upgrade code needs to be written once the changes are approved
4. There might be some simpler things we could do to improve things as a follow on. One example @keith-turner mentioned was maybe instead of deleting tablets during deletion we could just remove the files and call merge later. We may also be able to simplify the metadata updates into 1 method/loop vs 2 for deletes. I plan to explore some of this going forward.
5. All of the changes will need to be merged/refactored into Elasticity branch.
This closes #1327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on pull request #3758: No chop merge and delete
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3758:
URL: https://github.com/apache/accumulo/pull/3758#issuecomment-1723838388
I just kicked off a full IT build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on pull request #3758: No chop merge and delete
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3758:
URL: https://github.com/apache/accumulo/pull/3758#issuecomment-1724017663
I am closing this and reopening with a new branch that squashes the commits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon commented on pull request #3758: No chop merge and delete
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3758:
URL: https://github.com/apache/accumulo/pull/3758#issuecomment-1724030731
This PR has been superseeded by #3761
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] cshannon closed pull request #3758: No chop merge and delete
Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon closed pull request #3758: No chop merge and delete
URL: https://github.com/apache/accumulo/pull/3758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org