You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Richard Pijnenburg (Jira)" <ji...@apache.org> on 2022/08/18 15:12:00 UTC
[jira] [Updated] (ATLAS-4659) Atlas in HA mode fails to get healthy
[ https://issues.apache.org/jira/browse/ATLAS-4659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Richard Pijnenburg updated ATLAS-4659:
--------------------------------------
Environment: Zookeeper 3.8.0, Cassandra 4.0.5, Solr 8.11.1 (was: Zookeeper 3.8.0)
> Atlas in HA mode fails to get healthy
> -------------------------------------
>
> Key: ATLAS-4659
> URL: https://issues.apache.org/jira/browse/ATLAS-4659
> Project: Atlas
> Issue Type: Bug
> Affects Versions: 3.0.0
> Environment: Zookeeper 3.8.0, Cassandra 4.0.5, Solr 8.11.1
> Reporter: Richard Pijnenburg
> Priority: Major
>
> We are trying to setup atlas with the HA functionality using zookeeper 3.8.0
> Relevant logs:
> {code:java}
> 2022-08-18 14:57:06,924 INFO - [main:] ~ Found matched server id id1 with host port: atlas-0.atlas-headless.atlas.svc.cluster.local:21000 (AtlasServerIdSelector:65)
> 2022-08-18 14:57:06,924 INFO - [main:] ~ Starting leader election for id1 (ActiveInstanceElectorService:112)
> 2022-08-18 14:57:06,933 INFO - [main:] ~ Leader latch started for id1. (ActiveInstanceElectorService:118)
> 2022-08-18 14:57:06,991 INFO - [main:] ~ AtlasJsonProvider() instantiated (AtlasJsonProvider:53)
> 2022-08-18 14:57:07,296 WARN - [main-EventThread:] ~ Server instance with server id id1 is elected as leader (ActiveInstanceElectorService:152)
> 2022-08-18 14:57:07,296 WARN - [main-EventThread:] ~ Instance becoming active from PASSIVE (ServiceState:88
>
> ———
>
> 2022-08-18 14:57:27,818 INFO - [main-EventThread:] ~ Reacting to active state: initializing Kafka consumers (NotificationHookConsumer:421)
> 2022-08-18 14:57:27,819 INFO - [main-EventThread:] ~ ==> KafkaNotification.createConsumers(notificationType=HOOK, numConsumers=1, autoCommitEnabled=false) (KafkaNotification:194)
> 2022-08-18 14:57:28,237 INFO - [main-EventThread:] ~ <== KafkaNotification.createConsumers(notificationType=HOOK, numConsumers=1, autoCommitEnabled=false) (KafkaNotification:234)
> 2022-08-18 14:57:28,402 INFO - [main-EventThread:] ~ ==> TaskManagement.instanceIsActive() (TaskManagement:94)
> 2022-08-18 14:57:28,402 INFO - [main-EventThread:] ~ TaskManagement: Started! (TaskManagement:196)
> 2022-08-18 14:57:28,479 INFO - [NotificationHookConsumer thread-0:] ~ [atlas-hook-consumer-thread]: Starting (Logging:66)
> 2022-08-18 14:57:28,481 INFO - [NotificationHookConsumer thread-0:] ~ ==> HookConsumer doWork() (NotificationHookConsumer$HookConsumer:540)
> 2022-08-18 14:57:28,483 INFO - [NotificationHookConsumer thread-0:] ~ Atlas Server is not ready. Waiting for 1000 milliseconds to retry... (NotificationHookConsumer$HookConsumer:940)
> 2022-08-18 14:57:28,485 INFO - [main-EventThread:] ~ TaskManagement: Found: 0: Tasks in pending state. (TaskManagement:212)
> 2022-08-18 14:57:28,485 INFO - [main-EventThread:] ~ <== TaskManagement.instanceIsActive() (TaskManagement:98)
> 2022-08-18 14:57:28,485 INFO - [main-EventThread:] ~ ==> IndexRecoveryService.instanceIsActive() (IndexRecoveryService:117)
> 2022-08-18 14:57:28,485 INFO - [main-EventThread:] ~ <== IndexRecoveryService.instanceIsActive() (IndexRecoveryService:121)
> 2022-08-18 14:57:28,486 INFO - [index-health-monitor:] ~ Index Health Monitor: Starting... (IndexRecoveryService$RecoveryThread:175)
> 2022-08-18 14:57:28,487 ERROR - [main-EventThread:] ~ Got exception while activating (ActiveInstanceElectorService:162)
> org.apache.atlas.exception.AtlasBaseException: ActiveInstanceState.update resulted in exception.
> at org.apache.atlas.web.service.ActiveInstanceState.update(ActiveInstanceState.java:119)
> at org.apache.atlas.web.service.ActiveInstanceElectorService.isLeader(ActiveInstanceElectorService.java:158)
> at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:702)
> at org.apache.curator.framework.recipes.leader.LeaderLatch$9.apply(LeaderLatch.java:698)
> at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
> at org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
> at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92)
> at org.apache.curator.framework.recipes.leader.LeaderLatch.setLeadership(LeaderLatch.java:697)
> at org.apache.curator.framework.recipes.leader.LeaderLatch.checkLeadership(LeaderLatch.java:575)
> at org.apache.curator.framework.recipes.leader.LeaderLatch.access$600(LeaderLatch.java:65)
> at org.apache.curator.framework.recipes.leader.LeaderLatch$7.processResult(LeaderLatch.java:626)
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:883)
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:653)
> at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:152)
> at org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:187)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:627)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED]
> at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:823)
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:432)
> at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkExists(CuratorFrameworkImpl.java:459)
> at org.apache.atlas.web.service.ActiveInstanceState.update(ActiveInstanceState.java:109)
> ... 16 more
> 2022-08-18 14:57:28,487 WARN - [main-EventThread:] ~ Server instance with server id id1 is removed as leader (ActiveInstanceElectorService:199)
> 2022-08-18 14:57:28,487 WARN - [main-EventThread:] ~ Instance becoming passive from BECOMING_ACTIVE (ServiceState:119)
> 2022-08-18 14:57:28,487 INFO - [main-EventThread:] ~ ==> IndexRecoveryService.instanceIsPassive() (IndexRecoveryService:126)
> 2022-08-18 14:57:28,487 INFO - [main-EventThread:] ~ Index Health Monitor: Shutdown: Starting... (IndexRecoveryService$RecoveryThread:196)
> 2022-08-18 14:57:28,487 INFO - [main-EventThread:] ~ Index Health Monitor: Shutdown: Done! (IndexRecoveryService$RecoveryThread:206)
> 2022-08-18 14:57:29,484 INFO - [NotificationHookConsumer thread-0:] ~ Atlas Server is not ready. Waiting for 1000 milliseconds to retry... (NotificationHookConsumer$HookConsumer:940)
> 2022-08-18 14:57:30,484 INFO - [NotificationHookConsumer thread-0:] ~ Atlas Server is not ready. Waiting for 1000 milliseconds to retry... (NotificationHookConsumer$HookConsumer:940) {code}
> Running Atlas in non ha mode works fine
> The zookeeper instance is also used for Cassandra and Solr and those don't seem to have any issues with Zookeeper.
> It's unclear from the logs where the actual issue is.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)