[GitHub] [accumulo] keith-turner commented on a change in pull request #2422: Initial implementation of my vision for implementation of Scan Servers.

[GitBox <gi...@apache.org>]

keith-turner commented on a change in pull request #2422:
URL: https://github.com/apache/accumulo/pull/2422#discussion_r790981594



##########
File path: core/src/main/java/org/apache/accumulo/core/clientImpl/TabletServerBatchReaderIterator.java
##########
@@ -497,26 +499,44 @@ private void doLookups(Map<String,Map<KeyExtent,List<Range>>> binnedRanges,
     for (final String tsLocation : locations) {
 
       final Map<KeyExtent,List<Range>> tabletsRanges = binnedRanges.get(tsLocation);
-      if (maxTabletsPerRequest == Integer.MAX_VALUE || tabletsRanges.size() == 1) {
-        QueryTask queryTask = new QueryTask(tsLocation, tabletsRanges, failures, receiver, columns);
-        queryTasks.add(queryTask);
+      if (options.isUseScanServer()) {
+        // Ignore the tablets location and find a scan server to use
+        ScanServerLocator ssl = context.getScanServerLocator();
+        tabletsRanges.forEach((k, v) -> {
+          try {
+            String location = ssl.reserveScanServer(new TabletIdImpl(k));

Review comment:
       This seems really expensive doing an RPC to ZK before a scan and an RPC to ZK after the scan to unreserve.  I was thinking we could possibly just select a scan server and try to use it and if its busy it would throw some sort of thrift exception that indicates its busy.  When the client gets the busy exception it will try another scan server, similar to the reservation failing here. However this does not require going to ZK and also the busy check on scan servers could probably be orders of magnitude faster than ZK as it would not require persisting data in the ZK edit log and getting ZK server side quorom for a write.
   
   One way a busy check could work on a scan server is that it has a fixed number of threads in a thread pool using a [SynchronousQueue](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/SynchronousQueue.html).  When a scan task is submitted to this thread pool and all threads are busy the thread pool should reject it.  When the thread pool rejection happens then we can turn around and throw the busy thrift exception. All of this should be really quick on the scan server.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org