You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Hemanth Yamijala <yh...@gmail.com> on 2016/04/08 15:53:00 UTC
Review Request 45929: ATLAS-628: Starting two Atlas instances at the
same time causes exceptions in HA mode.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/
-----------------------------------------------------------
Review request for atlas.
Bugs: ATLAS-628
https://issues.apache.org/jira/browse/ATLAS-628
Repository: atlas
Description
-------
Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
Diffs
-----
distro/src/bin/atlas_start.py 7db74c0
repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/45929/diff/
Testing
-------
Existing ITs, UTs pass.
* Manually verified setting up with both embedded HBase setup and external HBase setup.
* Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
Thanks,
Hemanth Yamijala
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
> On April 15, 2016, 6:17 p.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java, line 43
> > <https://reviews.apache.org/r/45929/diff/3/?file=1344545#file1344545line43>
> >
> > TitanGraphProvider has some RunTimeExceptions. We could change that to SetupException. Also not sure what Titan throws when graph schema errors occur, what do they throw..should we wrap them in this?
I am catching the exceptions thrown from here in the GraphSchemaInitializer's setup and throwing it back as a SetupException. I also checked the calls from Titan when schema error occurs - they are mostly RuntimeExceptions - NullPointerException, IllegalArgumentException, etc. So, having a catch-all and rethrowing it as a SetupException will handle those.
- Hemanth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129165
-----------------------------------------------------------
On April 14, 2016, 3:24 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 3:24 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129165
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java (line 43)
<https://reviews.apache.org/r/45929/#comment192627>
TitanGraphProvider has some RunTimeExceptions. We could change that to SetupException. Also not sure what Titan throws when graph schema errors occur, what do they throw..should we wrap them in this?
- Suma Shivaprasad
On April 14, 2016, 3:24 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 3:24 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129663
-----------------------------------------------------------
Ship it!
Ship It!
- Suma Shivaprasad
On April 18, 2016, 6:31 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 6:31 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/ActiveInstanceState.java 6b39970
> webapp/src/main/java/org/apache/atlas/web/service/AtlasZookeeperSecurityProperties.java 40a6030
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/AtlasZookeeperSecurityPropertiesTest.java 118587a
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
> On April 20, 2016, 1 a.m., Suma Shivaprasad wrote:
> > webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java, line 93
> > <https://reviews.apache.org/r/45929/diff/3-4/?file=1344552#file1344552line93>
> >
> > We need to add some documentation for this as part of Installation docs?.
Added documentation in InstallationSteps.twiki and Configuration.twiki.
- Hemanth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129660
-----------------------------------------------------------
On April 20, 2016, 3:28 a.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 20, 2016, 3:28 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> docs/src/site/twiki/InstallationSteps.twiki 0851d53
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/ActiveInstanceState.java 6b39970
> webapp/src/main/java/org/apache/atlas/web/service/AtlasZookeeperSecurityProperties.java 40a6030
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/AtlasZookeeperSecurityPropertiesTest.java 118587a
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129660
-----------------------------------------------------------
webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java (line 92)
<https://reviews.apache.org/r/45929/#comment193157>
We need to add some documentation for this as part of Installation docs?.
- Suma Shivaprasad
On April 18, 2016, 6:31 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 6:31 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/ActiveInstanceState.java 6b39970
> webapp/src/main/java/org/apache/atlas/web/service/AtlasZookeeperSecurityProperties.java 40a6030
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/AtlasZookeeperSecurityPropertiesTest.java 118587a
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/
-----------------------------------------------------------
(Updated April 20, 2016, 3:28 a.m.)
Review request for atlas.
Changes
-------
Updates to documentation as per review comments.
Bugs: ATLAS-628
https://issues.apache.org/jira/browse/ATLAS-628
Repository: atlas
Description
-------
Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
Diffs (updated)
-----
distro/src/bin/atlas_start.py 7db74c0
distro/src/conf/atlas-application.properties 4131240
docs/src/site/twiki/Configuration.twiki 023f5a0
docs/src/site/twiki/InstallationSteps.twiki 0851d53
repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
webapp/src/main/java/org/apache/atlas/web/service/ActiveInstanceState.java 6b39970
webapp/src/main/java/org/apache/atlas/web/service/AtlasZookeeperSecurityProperties.java 40a6030
webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/AtlasZookeeperSecurityPropertiesTest.java 118587a
webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/45929/diff/
Testing
-------
Existing ITs, UTs pass.
* Manually verified setting up with both embedded HBase setup and external HBase setup.
* Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
Thanks,
Hemanth Yamijala
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/
-----------------------------------------------------------
(Updated April 18, 2016, 6:31 p.m.)
Review request for atlas.
Changes
-------
Addresses review comments from Suma. Responses to the comments are provided inline.
Bugs: ATLAS-628
https://issues.apache.org/jira/browse/ATLAS-628
Repository: atlas
Description
-------
Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
Diffs (updated)
-----
distro/src/bin/atlas_start.py 7db74c0
distro/src/conf/atlas-application.properties 4131240
docs/src/site/twiki/Configuration.twiki 023f5a0
repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
webapp/src/main/java/org/apache/atlas/web/service/ActiveInstanceState.java 6b39970
webapp/src/main/java/org/apache/atlas/web/service/AtlasZookeeperSecurityProperties.java 40a6030
webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/AtlasZookeeperSecurityPropertiesTest.java 118587a
webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/45929/diff/
Testing
-------
Existing ITs, UTs pass.
* Manually verified setting up with both embedded HBase setup and external HBase setup.
* Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
Thanks,
Hemanth Yamijala
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Suma Shivaprasad <su...@gmail.com>.
> On April 15, 2016, 6:31 p.m., Suma Shivaprasad wrote:
> > webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java, line 59
> > <https://reviews.apache.org/r/45929/diff/3/?file=1344552#file1344552line59>
> >
> > what is the expected behaviour in unclean shutdown case when lock is not cleared? Is this an ephemeral node that will be cleared automatically?
>
> Hemanth Yamijala wrote:
> I verified using Zookeeper Shell. The Lock nodes created by Curator are ephemeral nodes. Hence, they are automatically cleared when the client connection to Zookeeper breaks. The more important question though is what happens if the setup is halfway through when this happens. To handle this case, I took this approach:
>
> * It is critical to not have setup run multiple times in inconsistent state as the results could be pretty much undefined.
> * Given the above, it is safer, although very inconvenient to refuse to run setup if something is detected as wrong. Manual fixing will be required in this extreme scenario.
>
> To handle this, I have introduced a marker znode in Zookeeper to indicate setup is in progress. This is deleted only when all setup steps are done. If this znode is found on entering the critical section, Setup will fail with SetupException and a message stating there was an unclean shutdown of setup and this needs to be fixed manually.
>
> Like I said, in cases where the node failed to be cleared even when all setup is done, this would be inconvenient. But the approach leans on safety rather than convenience. Please let me know if this is OK. Leaving the issue open until review is done.
Yes this makes sense since the user would get to know that there was an unclean shutdown during setup which could have caused other issues like disabling the hbase 'titan' table which means the next setup could fail. Should we add this as as a known issue in documentation - the hbase setup ?
- Suma
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129171
-----------------------------------------------------------
On April 18, 2016, 6:31 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 18, 2016, 6:31 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/ActiveInstanceState.java 6b39970
> webapp/src/main/java/org/apache/atlas/web/service/AtlasZookeeperSecurityProperties.java 40a6030
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/AtlasZookeeperSecurityPropertiesTest.java 118587a
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
> On April 15, 2016, 6:31 p.m., Suma Shivaprasad wrote:
> > webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java, line 59
> > <https://reviews.apache.org/r/45929/diff/3/?file=1344552#file1344552line59>
> >
> > what is the expected behaviour in unclean shutdown case when lock is not cleared? Is this an ephemeral node that will be cleared automatically?
>
> Hemanth Yamijala wrote:
> I verified using Zookeeper Shell. The Lock nodes created by Curator are ephemeral nodes. Hence, they are automatically cleared when the client connection to Zookeeper breaks. The more important question though is what happens if the setup is halfway through when this happens. To handle this case, I took this approach:
>
> * It is critical to not have setup run multiple times in inconsistent state as the results could be pretty much undefined.
> * Given the above, it is safer, although very inconvenient to refuse to run setup if something is detected as wrong. Manual fixing will be required in this extreme scenario.
>
> To handle this, I have introduced a marker znode in Zookeeper to indicate setup is in progress. This is deleted only when all setup steps are done. If this znode is found on entering the critical section, Setup will fail with SetupException and a message stating there was an unclean shutdown of setup and this needs to be fixed manually.
>
> Like I said, in cases where the node failed to be cleared even when all setup is done, this would be inconvenient. But the approach leans on safety rather than convenience. Please let me know if this is OK. Leaving the issue open until review is done.
>
> Suma Shivaprasad wrote:
> Yes this makes sense since the user would get to know that there was an unclean shutdown during setup which could have caused other issues like disabling the hbase 'titan' table which means the next setup could fail. Should we add this as as a known issue in documentation - the hbase setup ?
Added documentation in InstallationSteps.twiki
- Hemanth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129171
-----------------------------------------------------------
On April 20, 2016, 3:28 a.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 20, 2016, 3:28 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> docs/src/site/twiki/InstallationSteps.twiki 0851d53
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/ActiveInstanceState.java 6b39970
> webapp/src/main/java/org/apache/atlas/web/service/AtlasZookeeperSecurityProperties.java 40a6030
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/AtlasZookeeperSecurityPropertiesTest.java 118587a
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
> On April 15, 2016, 6:31 p.m., Suma Shivaprasad wrote:
> > webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java, line 59
> > <https://reviews.apache.org/r/45929/diff/3/?file=1344552#file1344552line59>
> >
> > what is the expected behaviour in unclean shutdown case when lock is not cleared? Is this an ephemeral node that will be cleared automatically?
I verified using Zookeeper Shell. The Lock nodes created by Curator are ephemeral nodes. Hence, they are automatically cleared when the client connection to Zookeeper breaks. The more important question though is what happens if the setup is halfway through when this happens. To handle this case, I took this approach:
* It is critical to not have setup run multiple times in inconsistent state as the results could be pretty much undefined.
* Given the above, it is safer, although very inconvenient to refuse to run setup if something is detected as wrong. Manual fixing will be required in this extreme scenario.
To handle this, I have introduced a marker znode in Zookeeper to indicate setup is in progress. This is deleted only when all setup steps are done. If this znode is found on entering the critical section, Setup will fail with SetupException and a message stating there was an unclean shutdown of setup and this needs to be fixed manually.
Like I said, in cases where the node failed to be cleared even when all setup is done, this would be inconvenient. But the approach leans on safety rather than convenience. Please let me know if this is OK. Leaving the issue open until review is done.
- Hemanth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129171
-----------------------------------------------------------
On April 14, 2016, 3:24 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 3:24 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/#review129171
-----------------------------------------------------------
webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java (line 59)
<https://reviews.apache.org/r/45929/#comment192631>
what is the expected behaviour in unclean shutdown case when lock is not cleared? Is this an ephemeral node that will be cleared automatically?
- Suma Shivaprasad
On April 14, 2016, 3:24 p.m., Hemanth Yamijala wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45929/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 3:24 p.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-628
> https://issues.apache.org/jira/browse/ATLAS-628
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py 7db74c0
> distro/src/conf/atlas-application.properties 4131240
> docs/src/site/twiki/Configuration.twiki 023f5a0
> repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
> server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
> webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/45929/diff/
>
>
> Testing
> -------
>
> Existing ITs, UTs pass.
>
> * Manually verified setting up with both embedded HBase setup and external HBase setup.
> * Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
>
>
> Thanks,
>
> Hemanth Yamijala
>
>
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/
-----------------------------------------------------------
(Updated April 14, 2016, 3:24 p.m.)
Review request for atlas.
Changes
-------
Added the ability to run the setup steps as part of Atlas server start for those who don't want to rely on an additional setup step. This is enabled by a configuration (disabled by default) if required.
Bugs: ATLAS-628
https://issues.apache.org/jira/browse/ATLAS-628
Repository: atlas
Description
-------
Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
Diffs (updated)
-----
distro/src/bin/atlas_start.py 7db74c0
distro/src/conf/atlas-application.properties 4131240
docs/src/site/twiki/Configuration.twiki 023f5a0
repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/Atlas.java 58c386d
webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/45929/diff/
Testing
-------
Existing ITs, UTs pass.
* Manually verified setting up with both embedded HBase setup and external HBase setup.
* Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
Thanks,
Hemanth Yamijala
Re: Review Request 45929: ATLAS-628: Starting two Atlas instances at
the same time causes exceptions in HA mode.
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45929/
-----------------------------------------------------------
(Updated April 12, 2016, 6 a.m.)
Review request for atlas.
Changes
-------
Added Zookeeper lock to allow setup to run exclusively.
Bugs: ATLAS-628
https://issues.apache.org/jira/browse/ATLAS-628
Repository: atlas
Description
-------
Implements capability in atlas to run a set of registered setup steps. Currently the only step is setting up the backend graph repository. The intent is to extend this later for setting up Kafka at least and maybe other things like Solr.
Diffs (updated)
-----
distro/src/bin/atlas_start.py 7db74c0
repository/src/main/java/org/apache/atlas/repository/graph/GraphSchemaInitializer.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupException.java PRE-CREATION
server-api/src/main/java/org/apache/atlas/setup/SetupStep.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/service/CuratorFactory.java 773e04e
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetup.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/AtlasSetupModule.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/setup/SetupSteps.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/setup/SetupStepsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/45929/diff/
Testing
-------
Existing ITs, UTs pass.
* Manually verified setting up with both embedded HBase setup and external HBase setup.
* Also ran atlas HA instances at exactly same time and verified the noted exception does not happen.
Thanks,
Hemanth Yamijala