You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Tom Beerbower <tb...@hortonworks.com> on 2016/05/24 17:42:39 UTC
Review Request 47782: Atlas should use external HBase and SOLR
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/
-----------------------------------------------------------
Review request for atlas and Hemanth Yamijala.
Bugs: ATLAS-823
https://issues.apache.org/jira/browse/ATLAS-823
Repository: atlas
Description
-------
Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
2. not suitable for production use - no HA, local-storage
Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
Diffs
-----
distro/src/bin/atlas_start.py cb869bb
distro/src/bin/atlas_stop.py a72c724
distro/src/test/python/scripts/TestMetadata.py 00ca3e2
docs/src/site/twiki/Configuration.twiki 879c06c
docs/src/site/twiki/InstallationSteps.twiki 006e1e8
Diff: https://reviews.apache.org/r/47782/diff/
Testing
-------
updated unit test
mvn clean test
Thanks,
Tom Beerbower
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135220
-----------------------------------------------------------
Ship it!
Since the default build works, committing this. Filed https://issues.apache.org/jira/browse/ATLAS-832 to track starting hbase and solr with atlas_start with 'embedded-services' profile
- Shwetha GS
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/
-----------------------------------------------------------
(Updated May 26, 2016, 7:14 p.m.)
Review request for atlas and Hemanth Yamijala.
Bugs: ATLAS-823
https://issues.apache.org/jira/browse/ATLAS-823
Repository: atlas
Description
-------
Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
2. not suitable for production use - no HA, local-storage
Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
Diffs (updated)
-----
distro/pom.xml 1ab092d
distro/src/bin/atlas_config.py 8f3479c
distro/src/bin/atlas_start.py cb869bb
distro/src/conf/atlas-application.properties 26f84e9
distro/src/conf/atlas-env.sh 4f1f46c
distro/src/main/assemblies/standalone-package.xml 09ff47d
docs/src/site/twiki/Configuration.twiki 879c06c
docs/src/site/twiki/InstallationSteps.twiki 006e1e8
Diff: https://reviews.apache.org/r/47782/diff/
Testing
-------
updated unit test
mvn clean test
Thanks,
Tom Beerbower
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On May 26, 2016, 6:04 p.m., Shwetha GS wrote:
> > distro/src/conf/hbase_solr-atlas-application.properties, line 2
> > <https://reviews.apache.org/r/47782/diff/2/?file=1395357#file1395357line2>
> >
> > Instead of adding new files, where configs can go out of sync. Can you use single file and use resource filtering to populate the values correctly. Have added ATLAS-823-modified.patch on the jira, check if it makes sense
>
> Tom Beerbower wrote:
> Yeah, the replacement values is better. One question with that... since you are specifying the values for both profiles (dist and embedded_services) which one wins if both profiles are used for the build? Should the default values (berkeley, elasticsearch) be specified at a global level and then just overridden for the embedded_services profile? I'll try it out and update the patch. Thanks.
Updated diff to included changes from your patch. Thanks.
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Shwetha GS <ss...@hortonworks.com>.
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected?
>
> Tom Beerbower wrote:
> You have to run atlas-env.sh first.
>
> Shwetha GS wrote:
> Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead of adding this as an extra manual step?
>
> Tom Beerbower wrote:
> I guess that it could but atlas-env.sh isn't new. It has always been a requirement to run it first.
From InstallationSteps.twiki:
atlas-env.sh has been added to the Atlas conf. This file can be used to set various environment
variables that you need for you services. In addition you can set any other environment
variables you might need. This file will be sourced by atlas scripts before any commands are
executed.
I think the intention was to run atlas-env.sh as part of atlas_start always, but was missed. Thats how it is even in hive:
bin$ grep -r hive-env.sh *
hive:if [ -f "${HIVE_CONF_DIR}/hive-env.sh" ]; then
hive: . "${HIVE_CONF_DIR}/hive-env.sh"
Your call if you want to address that as part of another jira
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected?
>
> Tom Beerbower wrote:
> You have to run atlas-env.sh first.
>
> Shwetha GS wrote:
> Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead of adding this as an extra manual step?
>
> Tom Beerbower wrote:
> I guess that it could but atlas-env.sh isn't new. It has always been a requirement to run it first.
>
> Shwetha GS wrote:
> From InstallationSteps.twiki:
> atlas-env.sh has been added to the Atlas conf. This file can be used to set various environment
> variables that you need for you services. In addition you can set any other environment
> variables you might need. This file will be sourced by atlas scripts before any commands are
> executed.
>
> I think the intention was to run atlas-env.sh as part of atlas_start always, but was missed. Thats how it is even in hive:
> bin$ grep -r hive-env.sh *
> hive:if [ -f "${HIVE_CONF_DIR}/hive-env.sh" ]; then
> hive: . "${HIVE_CONF_DIR}/hive-env.sh"
>
> Your call if you want to address that as part of another jira
>
> Shwetha GS wrote:
> Running atlas-env.sh before atlas_start didn't start hbase and solr:
> apache-atlas-0.7-incubating-SNAPSHOT$ ./conf/atlas-env.sh
> apache-atlas-0.7-incubating-SNAPSHOT$ ./bin/atlas_start.py
> Apache Atlas Server started!!!
This works ...
>>source ./conf/atlas-env.sh
>>echo $MANAGE_LOCAL_HBASE
true
>>echo $MANAGE_LOCAL_SOLR
true
>>bin/atlas_start.py
configured for local hbase.
hbase started.
configured for local solr.
solr started.
setting up solr collections...
Apache Atlas Server started!!!
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected?
>
> Tom Beerbower wrote:
> You have to run atlas-env.sh first.
>
> Shwetha GS wrote:
> Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead of adding this as an extra manual step?
I guess that it could but atlas-env.sh isn't new. It has always been a requirement to run it first.
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Shwetha GS <ss...@hortonworks.com>.
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected?
>
> Tom Beerbower wrote:
> You have to run atlas-env.sh first.
>
> Shwetha GS wrote:
> Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead of adding this as an extra manual step?
>
> Tom Beerbower wrote:
> I guess that it could but atlas-env.sh isn't new. It has always been a requirement to run it first.
>
> Shwetha GS wrote:
> From InstallationSteps.twiki:
> atlas-env.sh has been added to the Atlas conf. This file can be used to set various environment
> variables that you need for you services. In addition you can set any other environment
> variables you might need. This file will be sourced by atlas scripts before any commands are
> executed.
>
> I think the intention was to run atlas-env.sh as part of atlas_start always, but was missed. Thats how it is even in hive:
> bin$ grep -r hive-env.sh *
> hive:if [ -f "${HIVE_CONF_DIR}/hive-env.sh" ]; then
> hive: . "${HIVE_CONF_DIR}/hive-env.sh"
>
> Your call if you want to address that as part of another jira
Running atlas-env.sh before atlas_start didn't start hbase and solr:
apache-atlas-0.7-incubating-SNAPSHOT$ ./conf/atlas-env.sh
apache-atlas-0.7-incubating-SNAPSHOT$ ./bin/atlas_start.py
Apache Atlas Server started!!!
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Shwetha GS <ss...@hortonworks.com>.
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected?
>
> Tom Beerbower wrote:
> You have to run atlas-env.sh first.
Can't atlas_start.py and atlas_stop.py in trun run atlas-env.sh, instead of adding this as an extra manual step?
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 7:14 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 7:14 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/atlas-env.sh 4f1f46c
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On May 26, 2016, 6:04 p.m., Shwetha GS wrote:
> > distro/src/conf/hbase_solr-atlas-application.properties, line 2
> > <https://reviews.apache.org/r/47782/diff/2/?file=1395357#file1395357line2>
> >
> > Instead of adding new files, where configs can go out of sync. Can you use single file and use resource filtering to populate the values correctly. Have added ATLAS-823-modified.patch on the jira, check if it makes sense
Yeah, the replacement values is better. One question with that... since you are specifying the values for both profiles (dist and embedded_services) which one wins if both profiles are used for the build? Should the default values (berkeley, elasticsearch) be specified at a global level and then just overridden for the embedded_services profile? I'll try it out and update the patch. Thanks.
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 2:17 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 2:17 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/hbase_solr-atlas-application.properties PRE-CREATION
> distro/src/conf/hbase_solr-atlas-env.sh PRE-CREATION
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
On May 26, 2016, 6:04 p.m., Tom Beerbower wrote:
> > In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected?
You have to run atlas-env.sh first.
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
On May 26, 2016, 2:17 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 2:17 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/hbase_solr-atlas-application.properties PRE-CREATION
> distro/src/conf/hbase_solr-atlas-env.sh PRE-CREATION
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review135028
-----------------------------------------------------------
distro/src/conf/hbase_solr-atlas-application.properties (line 2)
<https://reviews.apache.org/r/47782/#comment200015>
Instead of adding new files, where configs can go out of sync. Can you use single file and use resource filtering to populate the values correctly. Have added ATLAS-823-modified.patch on the jira, check if it makes sense
In the package built using '-Pdist,embedded_services', atlas_start doesn't start embedded hbase and solr. is that expected?
- Shwetha GS
On May 26, 2016, 2:17 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 2:17 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/hbase_solr-atlas-application.properties PRE-CREATION
> distro/src/conf/hbase_solr-atlas-env.sh PRE-CREATION
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/
-----------------------------------------------------------
(Updated May 26, 2016, 2:17 p.m.)
Review request for atlas and Hemanth Yamijala.
Changes
-------
Updated the diff so that the HBase and SOLR packaging are moved to a new embedded_services profile. By default the packaging is skipped and the configuration is set of Berkeley and ElasticSearch. If the embedded_services profile is used to build the distro then HBase and Solr are included and Atlas is configured to start and use them.
Bugs: ATLAS-823
https://issues.apache.org/jira/browse/ATLAS-823
Repository: atlas
Description
-------
Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
2. not suitable for production use - no HA, local-storage
Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
Diffs (updated)
-----
distro/pom.xml 1ab092d
distro/src/bin/atlas_config.py 8f3479c
distro/src/bin/atlas_start.py cb869bb
distro/src/conf/atlas-application.properties 26f84e9
distro/src/conf/hbase_solr-atlas-application.properties PRE-CREATION
distro/src/conf/hbase_solr-atlas-env.sh PRE-CREATION
distro/src/main/assemblies/standalone-package.xml 09ff47d
docs/src/site/twiki/Configuration.twiki 879c06c
docs/src/site/twiki/InstallationSteps.twiki 006e1e8
Diff: https://reviews.apache.org/r/47782/diff/
Testing
-------
updated unit test
mvn clean test
Thanks,
Tom Beerbower
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On May 25, 2016, 10:28 a.m., Shwetha GS wrote:
> > 1. Change distro/atlas-application.properties to use berkleydb and elasticsearch
> > 2. Remove packaging of hbase and solr
So we want to completely revert the embedded HBase and SOLR patches?
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review134747
-----------------------------------------------------------
On May 24, 2016, 5:42 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 24, 2016, 5:42 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py cb869bb
> distro/src/bin/atlas_stop.py a72c724
> distro/src/test/python/scripts/TestMetadata.py 00ca3e2
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Shwetha GS <ss...@hortonworks.com>.
> On May 25, 2016, 10:28 a.m., Shwetha GS wrote:
> > 1. Change distro/atlas-application.properties to use berkleydb and elasticsearch
> > 2. Remove packaging of hbase and solr
>
> Tom Beerbower wrote:
> So we want to completely revert the embedded HBase and SOLR patches?
>
> Madhan Neethiraj wrote:
> Yes, I think we should remove HBase and Solr from packaging.
Alternatively, we can move the packaging to another profile. But we should revert the config changes, so that default packaging with default configs works
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review134747
-----------------------------------------------------------
On May 24, 2016, 5:42 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 24, 2016, 5:42 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py cb869bb
> distro/src/bin/atlas_stop.py a72c724
> distro/src/test/python/scripts/TestMetadata.py 00ca3e2
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Madhan Neethiraj <ma...@apache.org>.
> On May 25, 2016, 10:28 a.m., Shwetha GS wrote:
> > 1. Change distro/atlas-application.properties to use berkleydb and elasticsearch
> > 2. Remove packaging of hbase and solr
>
> Tom Beerbower wrote:
> So we want to completely revert the embedded HBase and SOLR patches?
Yes, I think we should remove HBase and Solr from packaging.
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review134747
-----------------------------------------------------------
On May 24, 2016, 5:42 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 24, 2016, 5:42 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py cb869bb
> distro/src/bin/atlas_stop.py a72c724
> distro/src/test/python/scripts/TestMetadata.py 00ca3e2
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Tom Beerbower <tb...@hortonworks.com>.
> On May 25, 2016, 10:28 a.m., Shwetha GS wrote:
> > 1. Change distro/atlas-application.properties to use berkleydb and elasticsearch
> > 2. Remove packaging of hbase and solr
>
> Tom Beerbower wrote:
> So we want to completely revert the embedded HBase and SOLR patches?
>
> Madhan Neethiraj wrote:
> Yes, I think we should remove HBase and Solr from packaging.
>
> Shwetha GS wrote:
> Alternatively, we can move the packaging to another profile. But we should revert the config changes, so that default packaging with default configs works
Updated the diff to move the packaging to a different profile. Berkeley and ElasticSearch are the defaults.
- Tom
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review134747
-----------------------------------------------------------
On May 26, 2016, 2:17 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 26, 2016, 2:17 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/pom.xml 1ab092d
> distro/src/bin/atlas_config.py 8f3479c
> distro/src/bin/atlas_start.py cb869bb
> distro/src/conf/atlas-application.properties 26f84e9
> distro/src/conf/hbase_solr-atlas-application.properties PRE-CREATION
> distro/src/conf/hbase_solr-atlas-env.sh PRE-CREATION
> distro/src/main/assemblies/standalone-package.xml 09ff47d
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>
Re: Review Request 47782: Atlas should use external HBase and SOLR
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47782/#review134747
-----------------------------------------------------------
1. Change distro/atlas-application.properties to use berkleydb and elasticsearch
2. Remove packaging of hbase and solr
- Shwetha GS
On May 24, 2016, 5:42 p.m., Tom Beerbower wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47782/
> -----------------------------------------------------------
>
> (Updated May 24, 2016, 5:42 p.m.)
>
>
> Review request for atlas and Hemanth Yamijala.
>
>
> Bugs: ATLAS-823
> https://issues.apache.org/jira/browse/ATLAS-823
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Use of embedded HBase and SOLR is not preferred due to multiple concerns, including:
>
> 1. lack of support by Ambari makes it difficult to handle kerberization, upgrade/migration, start/stop
> 2. not suitable for production use - no HA, local-storage
>
> Atlas should remove support for embedded HBase and SOLR. Instead Atlas should be updated to use external HBase and SOLR.
>
>
> Diffs
> -----
>
> distro/src/bin/atlas_start.py cb869bb
> distro/src/bin/atlas_stop.py a72c724
> distro/src/test/python/scripts/TestMetadata.py 00ca3e2
> docs/src/site/twiki/Configuration.twiki 879c06c
> docs/src/site/twiki/InstallationSteps.twiki 006e1e8
>
> Diff: https://reviews.apache.org/r/47782/diff/
>
>
> Testing
> -------
>
> updated unit test
>
> mvn clean test
>
>
> Thanks,
>
> Tom Beerbower
>
>