You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Diane Hardman (JIRA)" <ji...@apache.org> on 2018/02/28 19:36:00 UTC
[jira] [Comment Edited] (GEODE-4451) When a gateway sender is
created using gfsh and fails to connect due to
AuthenticationFailedException, it is left in an inconsistent state
[ https://issues.apache.org/jira/browse/GEODE-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350828#comment-16350828 ]
Diane Hardman edited comment on GEODE-4451 at 2/28/18 7:35 PM:
---------------------------------------------------------------
If the gateway sender fails authentication with the receiver, the exception should be thrown and the gateway sender should retry to connection.
was (Author: dhardman):
If the gateway sender fails authentication with the receiver, the exception should be thrown and the gateway sender should be destroyed.
> When a gateway sender is created using gfsh and fails to connect due to AuthenticationFailedException, it is left in an inconsistent state
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-4451
> URL: https://issues.apache.org/jira/browse/GEODE-4451
> Project: Geode
> Issue Type: Bug
> Components: wan
> Reporter: Barry Oglesby
> Priority: Major
>
> If I attempt to create a gateway sender with invalid credentials using gfsh, it fails like:
> {noformat}
> (2) Executing - create gateway-sender --remote-distributed-system-id=1 --id=ny --dispatcher-threads=1 --parallel=true
> Member | Status
> ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> ln-1 | ERROR: Could not start a gateway sender ny because of exception Could not start a gateway sender 84 because of exception org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
> {noformat}
> The 84 should be the gateway sender id; instead its the thread id (from \{{ConcurrentParallelGatewaySenderEventProcessor.waitForRunningStatus}}.
> In the server log, I see 5 AuthenticationFailedException stacks like (one per dispatcher-thread):
> {noformat}
> [severe 2018/01/31 16:13:12.638 PST ln-1 <Event Processor for GatewaySender_ny_0> tid=0x55] Message dispatch failed due to unexpected exception..
> org.apache.geode.internal.cache.wan.GatewaySenderException: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials., caused by org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:432)
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.<init>(GatewaySenderEventRemoteDispatcher.java:75)
> at org.apache.geode.internal.cache.wan.parallel.RemoteParallelGatewaySenderEventProcessor.initializeEventDispatcher(RemoteParallelGatewaySenderEventProcessor.java:74)
> at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.setRunningStatus(AbstractGatewaySenderEventProcessor.java:1078)
> at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1050)
> Caused by: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
> at org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397)
> at org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253)
> at org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118)
> at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136)
> at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259)
> at org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206)
> at org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902)
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:388)
> ... 4 more
> {noformat}
> A thread dump shows the Event Processor threads are gone, but 5 BatchRemovalThreads like this exist:
> {noformat}
> "BatchRemovalThread" #96 daemon prio=10 os_prio=31 tid=0x00007fc1dd009000 nid=0xcb03 waiting on condition [0x000070000e245000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007a04043c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
> at org.apache.geode.internal.util.concurrent.StoppableCondition.await(StoppableCondition.java:61)
> at org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue$BatchRemovalThread.run(ParallelGatewaySenderQueue.java:1613)
> {noformat}
> list gateways shows the sender in the \{{Not Running}} state:
> {noformat}
> GatewaySender Id | Member | Remote Cluster Id | Type | Status | Queued Events | Receiver Location
> ---------------- | -------------------------------- | ----------------- | -------- | ----------- | ------------- | -----------------
> ny | 192.168.2.4(ln-1:13883)<v3>:1027 | 1 | Parallel | Not Running | 0 | null
> {noformat}
> The sender is left in an inconsistent state. It should either be completely gone or retrying to connect.
> {\{GatewaySenderEventRemoteDispatcher.<init>}} catches the \{{GatewaySenderException}} attempting to \{{initializeConnection}} and rethrows it if is caused by \{{GemFireSecurityException}}:
> {noformat}
> public GatewaySenderEventRemoteDispatcher(AbstractGatewaySenderEventProcessor eventProcessor) {
> this.processor = eventProcessor;
> this.sender = eventProcessor.getSender();
> // this.ackReaderThread = new AckReaderThread(sender);
> try {
> initializeConnection();
> } catch (GatewaySenderException e) {
> if (e.getCause() instanceof GemFireSecurityException) {
> throw e;
> }
> }
> }
> {noformat}
> {\{AbstractGatewaySenderEventProcessor.run}} catches that exception and logs severe message and stack above.
> If I change this area of code to log the warning instead of stopping the Event Processor threads, I see this exception after puts are done:
> {noformat}
> [severe 2018/01/31 17:37:16.324 PST ln-1 <Event Processor for GatewaySender_ny_0> tid=0x49] Stopping the processor because the following exception occurred while processing a batch:
> org.apache.geode.internal.cache.wan.GatewaySenderException: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials., caused by org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:431)
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.getConnection(GatewaySenderEventRemoteDispatcher.java:320)
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher._dispatchBatch(GatewaySenderEventRemoteDispatcher.java:198)
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.dispatchBatch(GatewaySenderEventRemoteDispatcher.java:147)
> at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.processQueue(AbstractGatewaySenderEventProcessor.java:609)
> at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1051)
> Caused by: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
> at org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397)
> at org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253)
> at org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118)
> at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136)
> at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259)
> at org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206)
> at org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902)
> at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:387)
> ... 5 more
> {noformat}
> That is coming from \{{GatewaySenderEventRemoteDispatcher.dispatchBatch}}:
> {noformat}
> } catch (GatewaySenderException ge) {
> if (...) {
> ...
> } else {
> logger.fatal(
> LocalizedMessage.create(
> LocalizedStrings.GatewayEventRemoteDispatcher_STOPPING_THE_PROCESSOR_BECAUSE_THE_FOLLOWING_EXCEPTION_OCCURRED_WHILE_PROCESSING_A_BATCH),
> ge);
> this.processor.setIsStopped(true);
> }
> {noformat}
> So, \{{AuthenticationFailedExceptions}} are definitely meant to stop the sender.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)