You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "ddanielr (via GitHub)" <gi...@apache.org> on 2023/08/15 18:05:50 UTC
[GitHub] [accumulo] ddanielr opened a new issue, #3694: Handle erroneously empty rfile block reads
ddanielr opened a new issue, #3694:
URL: https://github.com/apache/accumulo/issues/3694
**Describe the bug**
When HDFS returns empty results for a rfile block that actually has data, Accumulo does not error or attempt additional validation of the rfile block. This can cause issues with metadata table reads as specific tablet contents fail to return.
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: 1.10.3
**Expected behavior**
Accumulo should be able to determine if a rfile content location is actually empty or if data should have been returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #3694: Handle erroneously empty rfile block reads
Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on issue #3694:
URL: https://github.com/apache/accumulo/issues/3694#issuecomment-1680203415
> The AccumuloGC queried the metadata table to find file references for deletion candidates. The metadata table scan returned `~tab:~pr` entries but failed to return any `srv` or `file` entries.
>
> Because of this, the GC concluded that no file references existed for a collection of deletion candidates and then removed files from HDFS that had valid references on the metadata table.
This implies that one measure to try to prevent this from happening again could be to remove the locality groups in the metadata, so the file entries are in the same locality group as the file entries. However, this comes with its own problems: First, you'd have to force a full major compaction on the metadata table to force this to take effect (and that could be costly... and the IO could stress whatever underlying filesystem or disk errors are causing the problem in the first place). Second, this could cause slower metadata performance. Third, there may still be weird behavior where file entries and prev row entries are in different RFile blocks or different HDFS blocks, and there could still be hidden errors when one block is read and not when the other is read... it's not clear to me how that would actually manifest itself, though (because it's not clear to me why HDFS errors are failing to propagate up into our code to begin with).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ddanielr commented on issue #3694: Handle erroneously empty rfile block reads
Posted by "ddanielr (via GitHub)" <gi...@apache.org>.
ddanielr commented on issue #3694:
URL: https://github.com/apache/accumulo/issues/3694#issuecomment-1679407074
> You are saying that HDFS is returning a zero byte file block and the RFile Reader does not fail?
The RFile Reader does not fail and the HDFS contents are currently speculated to be empty.
Dmesg reports IO errors for the disk that hosted the rfile block.
>Does the DataNode log any error in this case?
The DataNode log does report `Slow flushOrSync` operations for the drive that contained the requested rfile block.
> What does fsck say about the file?
Currently unknown
> Is this a transient issue?
Yes, on subsequent reads, the rfile contents are returned correctly.
> Are other copies of the block correct?
Unknown, but is speculated that HDFS did not attempt to query the other block locations.
This issue is an attempt to chase down a transient bug with the AccumuloGC.
For this situation, the metadata table is separated into different locality groups `groupA: loc,~tab`, `groupB: file,log,srv,future`.
The tserver that hosted the groupB locality group reported disk errors.
The AccumuloGC queried the metadata table to find file references for deletion candidates.
The metadata table scan returned `~tab:~pr` entries but failed to return any `srv` or `file` entries.
Because of this, the GC concluded that no file references existed for a collection of deletion candidates and then removed files from HDFS that had valid references on the metadata table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] keith-turner commented on issue #3694: Handle erroneously empty rfile block reads
Posted by "keith-turner (via GitHub)" <gi...@apache.org>.
keith-turner commented on issue #3694:
URL: https://github.com/apache/accumulo/issues/3694#issuecomment-1682998430
I was reviewing some of the 2.1 Rfile code today and I realized this issue happened in 1.10. That made me diff the 1.10 and 2.1 code and I found that the fixes for #3617 are not in 1.10. Not sure if that matters, but just recording the finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] dlmarion commented on issue #3694: Handle erroneously empty rfile block reads
Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on issue #3694:
URL: https://github.com/apache/accumulo/issues/3694#issuecomment-1679350820
Does the DataNode log any error in this case? You are saying that HDFS is returning a zero byte file block and the RFile Reader does not fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org