[GitHub] [accumulo] billoley opened a new issue, #2789: Exception during iterator yield caused by multiple issues in LookupTask

[GitBox <gi...@apache.org>]

billoley opened a new issue, #2789:
URL: https://github.com/apache/accumulo/issues/2789

   Running version 1.9.3, observed an exception in the tserver logs from Tablet (now in TabletBase) --> "Underlying iterator yielded to a position that does not follow the last key returned: "  The exception didn't make sense because the last key returned (results.get(results.size() - 1)) was not in the seek range.  While writing a unit test to reproduce the issue, three separate issues in LookupTask were discovered.
   
   1) 
   
   In LookupTask.run(), session.queries is a HashMap (therefore not sorted), so when List<KVEntry> results = new ArrayList<>(); which contains the accumulated results for all subsequent KeyExtents gets passed to successive tablets, a yield could easily happen at a key that is before the last entry in results. ("Underlying iterator yielded to a position that does not follow the last key returned: ")  LookupTask.run() should pass an empty List of results and add the returned value to the accumulated results.
   
   2) 
   
   LookupTask.run() uses an iterator (Iterator<Entry<KeyExtent,List<Range>>> iter = session.queries.entrySet().iterator()). If there are unfinished ranges on a tablet that is not closed, it calls session.queries.put(entry.getKey(), lookupResult.unfinishedRanges) which is modifying the container that is being iterated over inside iteration loop.  This results in a ConcurrentModificationException the next time iter.next() is called.
   
   3) 
   
   After #1 and #2 are corrected and the unfinished ranges are re-added to session.queries for a subsequent call, there are issues with partScan/partNextKey/partNextKeyInclusive
   
   a) These variables are not used until after the loop exits, so they will be overwritten any time there is a lookupResult with unfinishedRanges on a tablet that is not closed.
   
   b) Once we complete scanning the extent/ranges that were previously unfinished, the last one encoutered (is this desired) will then be in partScan/partNextKey/partNextKeyInclusive and fullscans.  Both are returned in the thrift response..... and this causes a NullPointerException in TabletServerBatchReaderIterator at line 554 in trackScanning()
   ListIterator<Range> iterator = unscanned.get(ke).listIterator()
   
   sequence of events:
   
   Map<KeyExtent,List<Range>> unscanned starts with all requested values
   remove all failures from unscanned
   remove all fullScans from unscanned
   then....
     if (scanResult.partScan != null) {
         KeyExtent ke = new KeyExtent(scanResult.partScan);
         Key nextKey = new Key(scanResult.partNextKey);
   
         ListIterator<Range> iterator = unscanned.get(ke).listIterator();
   
   NullPointerException because the scan on ke was completed and it is in fullScans, thus removed from the unscanned container
   
   
   Encountered in version 1.9.3, but these issues exist in much older versions up to the current.
   
   Reproduced by adding another test to YieldScannersIt that added splits to the table prior to adding data, ensuring that LookupTask received a request that included more than one KeyExtent -> List<Range> entries.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] ctubbsii commented on issue #2789: Exception during iterator yield caused by multiple issues in LookupTask

[GitBox <gi...@apache.org>]

ctubbsii commented on issue #2789:
URL: https://github.com/apache/accumulo/issues/2789#issuecomment-1206545443

   Updated the original description to use markdown a bit better. In particular, there were issues displaying the generic parameters for the collections classes, because `<` and `>` are reserved characters in markdown, as in HTML, and need to be escaped using `&lt;` and `&gt;` or quoted using backticks to create a literal code block, as in ``` `List<String>` ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] ivakegg closed issue #2789: Exception during iterator yield caused by multiple issues in LookupTask

[GitBox <gi...@apache.org>]

ivakegg closed issue #2789: Exception during iterator yield caused by multiple issues in LookupTask
URL: https://github.com/apache/accumulo/issues/2789


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org