You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Nigel Jones <jo...@uk.ibm.com> on 2017/12/01 00:18:56 UTC
Re: Review Request 64141: [ATLAS-2287]: Include lucene libraries when
building atlas distribution with Janus profile
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64141/#review192410
-----------------------------------------------------------
I tried the new patch (2287.2) - mvn command as above (janus).
Build is clean, but I still get errors when starting the server in application.log, the UI is now giving me error 503. It still looks like zookeeper & hbase issues
2017-12-01 00:14:58,069 WARN - [main-SendThread(localhost:2181):] ~ Session 0x0 for server null, unexpected error, closing socket connecti
on and attempting reconnect (ClientCnxn$SendThread:1102)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Also
2017-12-01 00:12:41,095 WARN - [main:] ~ Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server (RecoverableZooKeeper:275)
2017-12-01 00:12:41,095 ERROR - [main:] ~ ZooKeeper getData failed after 4 attempts (RecoverableZooKeeper:277)
2017-12-01 00:12:41,096 WARN - [main:] ~ hconnection-0x23b8d9f30x0, quorum=localhost:2181, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server (ZKUtil:753)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:482)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:600)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
....
And some of these:
2017-12-01 00:12:41,096 ERROR - [main:] ~ hconnection-0x23b8d9f30x0, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperExce
ption, re-throwing exception (ZooKeeperWatcher:648)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:482)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:600)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
- Nigel Jones
On Nov. 30, 2017, 8:59 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64141/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2017, 8:59 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
>
>
> Bugs: ATLAS-2287
> https://issues.apache.org/jira/browse/ATLAS-2287
>
>
> Repository: atlas
>
>
> Description
> -------
>
> When Atlas is build using -Pdist profile, lucene jars are excluded during packaging of the war file. Since we are not shading graphdb module for janus profile, these jars are needed as run time dependency.
> Titan's shaded jar includes the lucene libraries and hence were excluded during packaging of war to avoid duplicate dependencies.
>
>
> Diffs
> -----
>
> distro/pom.xml eea256d8
> graphdb/janus/src/test/resources/atlas-application.properties ebc74ff3
> intg/src/test/resources/atlas-application.properties ce9351a6
> pom.xml 3720c1f5
> repository/pom.xml e24122e5
> webapp/pom.xml b4a96d36
>
>
> Diff: https://reviews.apache.org/r/64141/diff/2/
>
>
> Testing
> -------
>
> validated building atlas distribution using both janus and titan0 profile. Atlas starts fine and basic functionalities working.
>
> mvn clean install -DskipTests -Pdist,embedded-hbase-solr
> mvn clean install -DskipTests -Pdist,embedded-hbase-solr -DGRAPH-PROVIDER=titan0
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 64141: [ATLAS-2287]: Include lucene libraries when building atlas distribution with Janus profile
Posted by Nigel Jones <ni...@gmail.com>.
On 2017-12-01 00:52, "Nigel Jones"<ni...@gmail.com> wrote:
> > atlas.server.ha.zookeeper.connect=localhost:2181
Reading the posts from earlier I then added
00:57 $ export MANAGE_LOCAL_SOLR=true
00:57 $ export MANAGE_LOCAL_HBASE=true
00:58 $ echo atlas.graphdb.backend=org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase >> conf/atlas-application.properties
With these done, Atlas starts up much more cleanly. I hadn't realised we needed elements of the prior script alongside the patch. I would expect the patch/fix to have add the necessary config so that the build 'would just work'.
The index creation (as noted) is taking a while, and of course when the atlas startup complete returns, the initial type setup is still processing. This eventually completes. I'm also able to run quick_start (slowly), with just the final DSL query failing
So much better..... :-)
At the very least we should update the build docs to clarify these required steps, but I do think the less manual steps we have the better?
Re: Review Request 64141: [ATLAS-2287]: Include lucene libraries when building atlas distribution with Janus profile
Posted by Nigel Jones <ni...@gmail.com>.
On 2017-12-01 00:31, Madhan Neethiraj <ma...@apache.org> wrote:
> To get rid of âConnection refusedâ WARN logs, please set the following property in conf/atlas-application.properties and start Atlas
> atlas.server.ha.zookeeper.connect=localhost:2181
This line was commented out. I uncommented, but having then restarted atlas, I get the same zookeeper errors
> Not sure what causes HBase error âConnectionLoss for /hbase/meta-region-serverâ in your environment though. By any chance, is there another zookeeper running in your env?
No other zookeeper/hadoop components, just 'desktop' apps intellij, gaiandb, postgres/mysql etc.
I see from the review board item that David has similar issues?
Note that I did get your prior script to run clean...
I am building on mac? I know you mentioned previously some of the test attempts were on linux
Re: Review Request 64141: [ATLAS-2287]: Include lucene libraries when
building atlas distribution with Janus profile
Posted by Madhan Neethiraj <ma...@apache.org>.
Nigel,
To get rid of “Connection refused” WARN logs, please set the following property in conf/atlas-application.properties and start Atlas
:
atlas.server.ha.zookeeper.connect=localhost:2181
Not sure what causes HBase error “ConnectionLoss for /hbase/meta-region-server” in your environment though. By any chance, is there another zookeeper running in your env?
Thanks,
Madhan
From: Nigel Jones <no...@reviews.apache.org> on behalf of Nigel Jones <jo...@uk.ibm.com>
Reply-To: Nigel Jones <jo...@uk.ibm.com>
Date: Thursday, November 30, 2017 at 4:19 PM
To: Apoorv Naik <na...@gmail.com>, Madhan Neethiraj <ma...@apache.org>, Ashutosh Mestry <am...@hortonworks.com>
Cc: Nigel Jones <jo...@uk.ibm.com>, atlas <de...@atlas.incubator.apache.org>, Sarath Subramanian <sa...@apache.org>
Subject: Re: Review Request 64141: [ATLAS-2287]: Include lucene libraries when building atlas distribution with Janus profile
This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64141/
I tried the new patch (2287.2) - mvn command as above (janus).
Build is clean, but I still get errors when starting the server in application.log, the UI is now giving me error 503. It still looks like zookeeper & hbase issues
2017-12-01 00:14:58,069 WARN - [main-SendThread(localhost:2181):] ~ Session 0x0 for server null, unexpected error, closing socket connecti
on and attempting reconnect (ClientCnxn$SendThread:1102)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Also
2017-12-01 00:12:41,095 WARN - [main:] ~ Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server (RecoverableZooKeeper:275)
2017-12-01 00:12:41,095 ERROR - [main:] ~ ZooKeeper getData failed after 4 attempts (RecoverableZooKeeper:277)
2017-12-01 00:12:41,096 WARN - [main:] ~ hconnection-0x23b8d9f30x0, quorum=localhost:2181, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server (ZKUtil:753)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:482)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:600)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
....
And some of these:
2017-12-01 00:12:41,096 ERROR - [main:] ~ hconnection-0x23b8d9f30x0, quorum=localhost:2181, baseZNode=/hbase Received unexpected KeeperExce
ption, re-throwing exception (ZooKeeperWatcher:648)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:745)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:482)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:600)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
- Nigel Jones
On November 30th, 2017, 8:59 p.m. GMT, Sarath Subramanian wrote:
Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj.
By Sarath Subramanian.
Updated Nov. 30, 2017, 8:59 p.m.Bugs: ATLAS-2287
Repository: atlas
Description
When Atlas is build using -Pdist profile, lucene jars are excluded during packaging of the war file. Since we are not shading graphdb module for janus profile, these jars are needed as run time dependency.Titan's shaded jar includes the lucene libraries and hence were excluded during packaging of war to avoid duplicate dependencies.
Testing
validated building atlas distribution using both janus and titan0 profile. Atlas starts fine and basic functionalities working.
mvn clean install -DskipTests -Pdist,embedded-hbase-solrmvn clean install -DskipTests -Pdist,embedded-hbase-solr -DGRAPH-PROVIDER=titan0
Diffs
distro/pom.xml (eea256d8)
graphdb/janus/src/test/resources/atlas-application.properties (ebc74ff3)
intg/src/test/resources/atlas-application.properties (ce9351a6)
pom.xml (3720c1f5)
repository/pom.xml (e24122e5)
webapp/pom.xml (b4a96d36)
View Diff