You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/03/15 15:19:24 UTC
[GitHub] [accumulo] milleruntime opened a new issue #2567: Data files in Tablet differ from in-memory data
milleruntime opened a new issue #2567:
URL: https://github.com/apache/accumulo/issues/2567
**Describe the bug**
I saw this error while running Randomwalk tests on the 2.1 main branch.
`Data files in 28;7;6 differ from in-memory data`
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: [2.1]
- OS: [Ubuntu]
**To Reproduce**
Steps to reproduce the behavior (or a link to an example repository that reproduces the problem):
1. Setup Uno. I had 2 tservers running locally.
2. Ingest some data for a while using Randomwalk MultiTable test
4. See error
Tserver log activity:
<pre>
2022-03-15T09:17:46,688 [memory.LargestFirstMemoryManager] DEBUG: COMPACTING !0;~< total = 203,913,354 ingestMemory = 22,659,626
2022-03-15T09:17:47,651 [memory.LargestFirstMemoryManager] DEBUG: COMPACTING 28;9;8 total = 226,570,072 ingestMemory = 45,313,436
2022-03-15T09:17:50,471 [tablet.Tablet] ERROR: Data files in 28;7;6 differ from in-memory data ...
2022-03-15T09:17:50,655 [tablet.files] DEBUG: Flushed 28;7;6 created hdfs://localhost:8020/accumulo/tables/28/t-00002pt/F00004hu.rf from [memory]
2022-03-15T09:17:50,657 [tablet.files] DEBUG: Flushed !0;~< created hdfs://localhost:8020/accumulo/tables/!0/table_info/F00004hz.rf from [memory]
2022-03-15T09:17:50,662 [tablet.files] DEBUG: Compacting !0;~< on i.meta.small for SYSTEM from [F00004hz.rf, A00004hy.rf] size 20 KB
2022-03-15T09:17:50,676 [memory.LargestFirstMemoryManager] DEBUG: COMPACTING 28;7;6 total = 272,062,261 ingestMemory = 90,808,517
2022-03-15T09:17:50,676 [memory.LargestFirstMemoryManager] DEBUG: chosenMem = 45,493,788 chosenIT = 2.31 load 45,574,937
2022-03-15T09:17:50,677 [tablet.Tablet] DEBUG: MinC initiate lock 0.00 secs
2022-03-15T09:17:51,540 [memory.LargestFirstMemoryManager] DEBUG: BEFORE compactionThreshold = 0.056 maxObserved = 339,854,955
2022-03-15T09:17:51,540 [memory.LargestFirstMemoryManager] DEBUG: AFTER compactionThreshold = 0.056
2022-03-15T09:17:51,596 [constraints.ConstraintChecker] DEBUG: Loaded constraint org.apache.accumulo.server.constraints.MetadataConstraints for !0
2022-03-15T09:17:51,611 [tablet.files] DEBUG: Flushed 28;9;8 created hdfs://localhost:8020/accumulo/tables/28/t-00002pv/F00004i0.rf from [memory]
2022-03-15T09:17:51,613 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:50726; action: authenticate;
2022-03-15T09:17:51,652 [tablet.files] DEBUG: Compacted !0;~< for SYSTEM created hdfs://localhost:8020/accumulo/tables/!0/table_info/A00004i2.rf from [F00004hz.rf, A00004hy.rf]
</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1070900506
It could be that we are just doing the check too often. We are currently calling it in 2 places:
- [Tablet.closeConsistencyCheck()](https://github.com/apache/accumulo/blob/6f8be5c9f26676712eb417db69ea1938380f4ff3/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java#L1422)
- TabletServer.run() -> [some checking thread()](https://github.com/apache/accumulo/blob/dca3ae8d056a24510aa9d86347d9d6338613c4b3/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L830)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1068145637
The error printed is a new consistency check added in 2.1. I think it was added in #2320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime edited a comment on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime edited a comment on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1070900506
It could be that we are just doing the check too often. We are currently calling it in 2 places:
- [Tablet.closeConsistencyCheck()](https://github.com/apache/accumulo/blob/6f8be5c9f26676712eb417db69ea1938380f4ff3/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java#L1422)
- TabletServer.run() -> [some checking thread()](https://github.com/apache/accumulo/blob/dca3ae8d056a24510aa9d86347d9d6338613c4b3/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L830)
Edit: It seems likes the error mostly occurs during the tserver thread check. I am doing more testing with RW trying to reproduce it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1069222310
> So, the tablet is responsible for handling the flush, so it is creating the file, so it should know about it already, but perhaps not in the "in-memory" structure that is being checked by this consistency check. It seems there's a deficiency in the consistency check and that check needs to be improved to include any other files the tablet knows about elsewhere in the tserver's in-memory structures.
Yeah it seems like the new consistency check could be better. I am not sure how to do that though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime edited a comment on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime edited a comment on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1070900506
It could be that we are just doing the check too often. We are currently calling it in 2 places:
- [Tablet.closeConsistencyCheck()](https://github.com/apache/accumulo/blob/6f8be5c9f26676712eb417db69ea1938380f4ff3/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java#L1422)
- TabletServer.run() -> [some checking thread()](https://github.com/apache/accumulo/blob/dca3ae8d056a24510aa9d86347d9d6338613c4b3/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L830)
Edit: It seems likes the error mostly occurs during the tserver thread check. I am doing more testing with RW trying to reproduce it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1069024735
> Have you been able to determine what circumstances cause them to be inconsistent?
Not exactly. It seems like this happened once resources got constrained. There is a chance it is related to this: https://github.com/apache/accumulo-testing/issues/196. I don't think it is a coincidence that the same table (multi_ip_10_113_12_25_17) that was the destination of the copyTool and took forever to copy was also the one that had this problem. The Copy started at `8:47:12`. It was taking a while but finished at `09:26:53`.
> Is it just routine after a compaction?
No, I wouldn't call it routine behavior but I don't know if it is a problem. That tablet flushed and compacted successfully multiple times before that error.
> Is it consistent eventually? If they aren't consistent, which side has the missing references?
The table seems to be functioning properly. But this isn't CI so I don't have a way to verify that no data was lost. I was thinking of adding a scanner node to the MultiTable test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1069175637
OK here is what was going on:
<pre>
2022-03-15T09:17:50,676 [memory.LargestFirstMemoryManager] DEBUG: COMPACTING 28;7;6 total = 272,062,261 ingestMemory = 90,808,517
tserver1.log:2022-03-15T09:17:50,471 [tablet.Tablet] ERROR: Data files in 28;7;6 differ from in-memory data ...
tserver1.log:2022-03-15T09:17:50,655 [tablet.files] DEBUG: Flushed 28;7;6 created hdfs://localhost:8020/accumulo/tables/28/t-00002pt/F00004hu.rf from [memory]
</pre>
The reason for the difference between in-memory and on disk is because the file (F00004hu.rf) had not been flushed yet. Looking at the difference in the logs between the 2 sets of files (there were 160 r-files in each set), the second set of files (memory) was missing the file. In other words, the file was on disk but not in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1068341990
Have you been able to determine what circumstances cause them to be inconsistent? Is it just routine after a compaction? Is it consistent eventually? If they aren't consistent, which side has the missing references?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1069228967
> > Yeah it seems like the new consistency check could be better. I am not sure how to do that though.
>
> I'm not sure either, but I would start by trying to figure out where in the tserver code does it already have a record about the file being flushed.
I pinged @keith-turner to see if he had any thoughts. I probably won't get to look at this until later in the week. I still think it should be a blocker though for 2.1. I would imagine this error happening a lot on a very active cluster. Plus the error sounds scary :fearful:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1070900506
It could be that we are just doing the check too often. We are currently calling it in 2 places:
- [Tablet.closeConsistencyCheck()](https://github.com/apache/accumulo/blob/6f8be5c9f26676712eb417db69ea1938380f4ff3/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Tablet.java#L1422)
- TabletServer.run() -> [some checking thread()](https://github.com/apache/accumulo/blob/dca3ae8d056a24510aa9d86347d9d6338613c4b3/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L830)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] milleruntime commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
milleruntime commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1068313033
Saw this error again with another table and with the metadata table. `Data files in !0;~< differ from in-memory data`
Is this a problem? I am not sure. Maybe it should just be a warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1069192364
So, the tablet is responsible for handling the flush, so it is creating the file, so it should know about it already, but perhaps not in the "in-memory" structure that is being checked by this consistency check. It seems there's a deficiency in the consistency check and that check needs to be improved to include any other files the tablet knows about elsewhere in the tserver's in-memory structures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on issue #2567: Data files in Tablet differ from in-memory data
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #2567:
URL: https://github.com/apache/accumulo/issues/2567#issuecomment-1069225905
> Yeah it seems like the new consistency check could be better. I am not sure how to do that though.
I'm not sure either, but I would start by trying to figure out where in the tserver code does it already have a record about the file being flushed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org