You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2021/06/30 15:11:24 UTC

[GitHub] [accumulo] milleruntime opened a new issue #2184: Thrift exception during upgrade

milleruntime opened a new issue #2184:
URL: https://github.com/apache/accumulo/issues/2184


   I was testing upgrade code for upgrading 2.0 to 2.1 and saw this exception quite a bit. 
   <pre>
   2021-06-30T10:45:31,675 [thrift.ProcessFunction] ERROR: Internal error processing reportTabletStatus
   org.apache.accumulo.core.clientImpl.thrift.ThriftNotActiveServiceException: null
           at org.apache.accumulo.server.rpc.HighlyAvailableServiceInvocationHandler.invoke(HighlyAvailableServiceInvocationHandler.java:53) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
           at com.sun.proxy.$Proxy38.reportTabletStatus(Unknown Source) ~[?:?]
           at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
           at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
           at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
           at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
           at org.apache.accumulo.core.trace.TraceUtil.lambda$wrapService$6(TraceUtil.java:235) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
           at com.sun.proxy.$Proxy38.reportTabletStatus(Unknown Source) ~[?:?]
           at org.apache.accumulo.core.manager.thrift.ManagerClientService$Processor$reportTabletStatus.getResult(ManagerClientService.java:1874) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
           at org.apache.accumulo.core.manager.thrift.ManagerClientService$Processor$reportTabletStatus.getResult(ManagerClientService.java:1855) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
           at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.12.0.jar:0.12.0]
           at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.12.0.jar:0.12.0]
           at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
           at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) ~[libthrift-0.12.0.jar:0.12.0]
           at org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.invoke(CustomNonBlockingServer.java:114) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
           at org.apache.thrift.server.Invocation.run(Invocation.java:18) ~[libthrift-0.12.0.jar:0.12.0]
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
           at java.lang.Thread.run(Thread.java:829) [?:?]
   </pre>
   
   From the description of `HighlyAvailableServiceWrapper` it looks like the class is doing its job. But I think the new upgrade code was added after this class was created so I am not sure if it should be running while the Manager is doing upgrades.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] Manno15 removed a comment on issue #2184: Thrift exception during upgrade

Posted by GitBox <gi...@apache.org>.
Manno15 removed a comment on issue #2184:
URL: https://github.com/apache/accumulo/issues/2184#issuecomment-934673924


   From my testing, the exception always occurs right after either the Root table or Metadata tablets would sleep during the upgrade process (`TabletGroupWatcher`: line 346). Though the sleep occurs more often than the exception so may be unrelated. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] Manno15 commented on issue #2184: Thrift exception during upgrade

Posted by GitBox <gi...@apache.org>.
Manno15 commented on issue #2184:
URL: https://github.com/apache/accumulo/issues/2184#issuecomment-934673924


   From my testing, the exception always occurs right after either the Root table or Metadata tablets would sleep (`TabletGroupWatcher`: line 346). Though the sleep occurs more often than the exception so may be unrelated. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] Manno15 edited a comment on issue #2184: Thrift exception during upgrade

Posted by GitBox <gi...@apache.org>.
Manno15 edited a comment on issue #2184:
URL: https://github.com/apache/accumulo/issues/2184#issuecomment-934673924


   From my testing, the exception always occurs right after either the Root table or Metadata tablets would sleep during the upgrade process (`TabletGroupWatcher`: line 346). Though the sleep occurs more often than the exception so may be unrelated. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] Manno15 closed issue #2184: Thrift exception during upgrade

Posted by GitBox <gi...@apache.org>.
Manno15 closed issue #2184:
URL: https://github.com/apache/accumulo/issues/2184


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] Manno15 removed a comment on issue #2184: Thrift exception during upgrade

Posted by GitBox <gi...@apache.org>.
Manno15 removed a comment on issue #2184:
URL: https://github.com/apache/accumulo/issues/2184#issuecomment-934673924


   From my testing, the exception always occurs right after either the Root table or Metadata tablets would sleep during the upgrade process (`TabletGroupWatcher`: line 346). Though the sleep occurs more often than the exception so may be unrelated. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] Manno15 commented on issue #2184: Thrift exception during upgrade

Posted by GitBox <gi...@apache.org>.
Manno15 commented on issue #2184:
URL: https://github.com/apache/accumulo/issues/2184#issuecomment-937323634


   The issue appears to be when the tablet server attempts to send a message to the manager during the upgrade process. The line in `TabletServer` and of the send function that calls `reportTabletStatus` is below.
   
   https://github.com/apache/accumulo/blob/1dc72fce2c781dee597c8c11876a3bc6c321c199/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L811
   
   https://github.com/apache/accumulo/blob/30ce59fd94f51b60a30ced328156f02d3223330b/server/tserver/src/main/java/org/apache/accumulo/tserver/managermessage/TabletStatusMessage.java#L40-L42


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] Manno15 commented on issue #2184: Thrift exception during upgrade

Posted by GitBox <gi...@apache.org>.
Manno15 commented on issue #2184:
URL: https://github.com/apache/accumulo/issues/2184#issuecomment-937323634


   The issue appears to be when the tablet server attempts to send a message to the manager during the upgrade process. The line in `TabletServer` and of the send function that calls `reportTabletStatus` is below.
   
   https://github.com/apache/accumulo/blob/1dc72fce2c781dee597c8c11876a3bc6c321c199/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L811
   
   https://github.com/apache/accumulo/blob/30ce59fd94f51b60a30ced328156f02d3223330b/server/tserver/src/main/java/org/apache/accumulo/tserver/managermessage/TabletStatusMessage.java#L40-L42


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org