You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Pierre Padovani <pi...@padovani.org> on 2018/04/12 17:35:44 UTC
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
-----------------------------------------------------------
(Updated April 12, 2018, 5:35 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This is the original patch. I swear I had uploaded it before.
Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478
Repository: atlas
Description (updated)
-------
This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
**Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd
distro/pom.xml 0103bef6
distro/src/bin/atlas_config.py e6415cf4
distro/src/bin/atlas_start.py 39be6b7c
distro/src/bin/atlas_stop.py 66edd904
distro/src/conf/atlas-env.sh 68b24e93
distro/src/main/assemblies/standalone-package.xml 1881082e
docs/src/site/twiki/Configuration.twiki 63c3fce9
docs/src/site/twiki/HighAvailability.twiki 4270d097
docs/src/site/twiki/InstallationSteps.twiki 6b9f0313
graphdb/janus/pom.xml 143b775f
pom.xml 34e1ad04
webapp/pom.xml 284f538f
Diff: https://reviews.apache.org/r/66064/diff/1/
Testing
-------
We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
Thanks,
Pierre Padovani
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On April 12, 2018, 11:49 p.m., Madhan Neethiraj wrote:
> > common/src/main/java/org/apache/atlas/repository/Constants.java
> > Line 56 (original), 56 (patched)
> > <https://reviews.apache.org/r/66064/diff/1/?file=1997842#file1997842line56>
> >
> > As you noted in the description, this would require re-indexing of existing deployments - which can be very expensive depending upon the amount of data. This should be avoided.
> >
> > If an index software has restrictions on the keynames, consider replacing these constants with calls to instance-specific property key provider. And implement a default one with current values and ElasticSearch specific one with necessary key names.
I believe I have a solution in place, but would like feedback on implementation.
> On April 12, 2018, 11:49 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 674 (original), 675 (patched)
> > <https://reviews.apache.org/r/66064/diff/1/?file=1997853#file1997853line675>
> >
> > Would this change cause elasticsearch libraries to be included for Solr deployments as well? If yes, please consider alternate approach (like a separate profile) for Elasticsearch - to avoid potential issues in deployments that use Solr.
> >
> > Same applies for line #694 below.
Only for the standard -Pdist profile. This shouldn't be an issue since the newer elasticsearch jars contain no shadded classes and should not conflict.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201061
-----------------------------------------------------------
On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 13, 2018, 2:55 a.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d
> distro/pom.xml 1f4c6d557
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 9213f488b
> distro/src/main/assemblies/standalone-package.xml dc2a66ba9
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 4d2ab7391
> webapp/pom.xml 284f538f7
>
>
> Diff: https://reviews.apache.org/r/66064/diff/2/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201061
-----------------------------------------------------------
common/src/main/java/org/apache/atlas/repository/Constants.java
Line 56 (original), 56 (patched)
<https://reviews.apache.org/r/66064/#comment282024>
As you noted in the description, this would require re-indexing of existing deployments - which can be very expensive depending upon the amount of data. This should be avoided.
If an index software has restrictions on the keynames, consider replacing these constants with calls to instance-specific property key provider. And implement a default one with current values and ElasticSearch specific one with necessary key names.
pom.xml
Line 674 (original), 675 (patched)
<https://reviews.apache.org/r/66064/#comment282022>
Would this change cause elasticsearch libraries to be included for Solr deployments as well? If yes, please consider alternate approach (like a separate profile) for Elasticsearch - to avoid potential issues in deployments that use Solr.
Same applies for line #694 below.
webapp/pom.xml
Lines 398 (patched)
<https://reviews.apache.org/r/66064/#comment282025>
Consider adding elastic-search libraries only for specific profiles.
- Madhan Neethiraj
On April 12, 2018, 5:35 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 12, 2018, 5:35 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd
> distro/pom.xml 0103bef6
> distro/src/bin/atlas_config.py e6415cf4
> distro/src/bin/atlas_start.py 39be6b7c
> distro/src/bin/atlas_stop.py 66edd904
> distro/src/conf/atlas-env.sh 68b24e93
> distro/src/main/assemblies/standalone-package.xml 1881082e
> docs/src/site/twiki/Configuration.twiki 63c3fce9
> docs/src/site/twiki/HighAvailability.twiki 4270d097
> docs/src/site/twiki/InstallationSteps.twiki 6b9f0313
> graphdb/janus/pom.xml 143b775f
> pom.xml 34e1ad04
> webapp/pom.xml 284f538f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/1/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the patch today and I am still having trouble getting Atlas up.
>
> Pierre Padovani wrote:
> Let me rebase the patch and make some changes, and retest. Master has moved a bit since I generated the original patch. I'll update as soon as I have it done.
>
> Pierre Padovani wrote:
> I updated the patch four days ago, can you please retest?
I will retest and let you know. Please give me a couple of days.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
-----------------------------------------------------------
On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 13, 2018, 2:55 a.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d
> distro/pom.xml 1f4c6d557
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 9213f488b
> distro/src/main/assemblies/standalone-package.xml dc2a66ba9
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 4d2ab7391
> webapp/pom.xml 284f538f7
>
>
> Diff: https://reviews.apache.org/r/66064/diff/2/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the patch today and I am still having trouble getting Atlas up.
Let me rebase the patch and make some changes, and retest. Master has moved a bit since I generated the original patch. I'll update as soon as I have it done.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
-----------------------------------------------------------
On April 12, 2018, 5:35 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 12, 2018, 5:35 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd
> distro/pom.xml 0103bef6
> distro/src/bin/atlas_config.py e6415cf4
> distro/src/bin/atlas_start.py 39be6b7c
> distro/src/bin/atlas_stop.py 66edd904
> distro/src/conf/atlas-env.sh 68b24e93
> distro/src/main/assemblies/standalone-package.xml 1881082e
> docs/src/site/twiki/Configuration.twiki 63c3fce9
> docs/src/site/twiki/HighAvailability.twiki 4270d097
> docs/src/site/twiki/InstallationSteps.twiki 6b9f0313
> graphdb/janus/pom.xml 143b775f
> pom.xml 34e1ad04
> webapp/pom.xml 284f538f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/1/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the patch today and I am still having trouble getting Atlas up.
>
> Pierre Padovani wrote:
> Let me rebase the patch and make some changes, and retest. Master has moved a bit since I generated the original patch. I'll update as soon as I have it done.
>
> Pierre Padovani wrote:
> I updated the patch four days ago, can you please retest?
>
> Ashutosh Mestry wrote:
> I will retest and let you know. Please give me a couple of days.
>
> Pierre Padovani wrote:
> Any updates on this?
>
> Thanks!
Can you please rebase this with latest master?
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
-----------------------------------------------------------
On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 13, 2018, 2:55 a.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d
> distro/pom.xml 1f4c6d557
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 9213f488b
> distro/src/main/assemblies/standalone-package.xml dc2a66ba9
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 4d2ab7391
> webapp/pom.xml 284f538f7
>
>
> Diff: https://reviews.apache.org/r/66064/diff/2/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the patch today and I am still having trouble getting Atlas up.
>
> Pierre Padovani wrote:
> Let me rebase the patch and make some changes, and retest. Master has moved a bit since I generated the original patch. I'll update as soon as I have it done.
I updated the patch four days ago, can you please retest?
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
-----------------------------------------------------------
On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 13, 2018, 2:55 a.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d
> distro/pom.xml 1f4c6d557
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 9213f488b
> distro/src/main/assemblies/standalone-package.xml dc2a66ba9
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 4d2ab7391
> webapp/pom.xml 284f538f7
>
>
> Diff: https://reviews.apache.org/r/66064/diff/2/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On April 13, 2018, 12:34 a.m., Ashutosh Mestry wrote:
> > Can you please add instructions on how to use this profile? I tried the patch today and I am still having trouble getting Atlas up.
>
> Pierre Padovani wrote:
> Let me rebase the patch and make some changes, and retest. Master has moved a bit since I generated the original patch. I'll update as soon as I have it done.
>
> Pierre Padovani wrote:
> I updated the patch four days ago, can you please retest?
>
> Ashutosh Mestry wrote:
> I will retest and let you know. Please give me a couple of days.
Any updates on this?
Thanks!
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
-----------------------------------------------------------
On April 13, 2018, 2:55 a.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 13, 2018, 2:55 a.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d
> distro/pom.xml 1f4c6d557
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 9213f488b
> distro/src/main/assemblies/standalone-package.xml dc2a66ba9
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 4d2ab7391
> webapp/pom.xml 284f538f7
>
>
> Diff: https://reviews.apache.org/r/66064/diff/2/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review201069
-----------------------------------------------------------
Can you please add instructions on how to use this profile? I tried the patch today and I am still having trouble getting Atlas up.
- Ashutosh Mestry
On April 12, 2018, 5:35 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated April 12, 2018, 5:35 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd
> distro/pom.xml 0103bef6
> distro/src/bin/atlas_config.py e6415cf4
> distro/src/bin/atlas_start.py 39be6b7c
> distro/src/bin/atlas_stop.py 66edd904
> distro/src/conf/atlas-env.sh 68b24e93
> distro/src/main/assemblies/standalone-package.xml 1881082e
> docs/src/site/twiki/Configuration.twiki 63c3fce9
> docs/src/site/twiki/HighAvailability.twiki 4270d097
> docs/src/site/twiki/InstallationSteps.twiki 6b9f0313
> graphdb/janus/pom.xml 143b775f
> pom.xml 34e1ad04
> webapp/pom.xml 284f538f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/1/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Madhan Neethiraj <ma...@apache.org>.
> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when building for solr?
>
> Pierre Padovani wrote:
> Yes... however the new 5.x version of the Elasticsearch jars no longer shade 3rd party libraries so are far safer than the earlier versions. So we have a choice, include the jars, or force users who wish to use Elasticsearch as the search backend to copy in the jars manually.
>
> Madhan Neethiraj wrote:
> Pierre - I would suggest to not package libraries from multiple search backends in Atlas. It should be done either via profiles or as you suggest by manually copying necessary libraries.
>
> Pierre Padovani wrote:
> Not to be obstinate about this... but have you tried to install a different set of libraries into the default atlas distribution? The lib directory doesn't exist, so you have to:
>
> -start atlas
> -have it unpack itself
> -fail to start
> -stop atlas
> -copy in the libraries needed
> -start atlas
>
> The above assumes the user isn't going to remove the solr jars... since they likely don't know about them. In the end it is very likely that they will end up with both search backend jars if they choose to use Elasticsearch.
>
> IMHO - This indicates that there is a bigger distribution packaging issue in place. What we should do is not include any backend or search jars in the default war build. Instead include them in the zip, and provide a setup script that allows the user to choose their setup which in turn unpacks and copies in the require jars.
>
> Finally, can you enlighten me as to the issue with including the jars in the default distribution? I understand it is your preference, but I fail to grasp what the conflict happens to be except for overall package size.
This removes existing exclusion of all elasicsearch libraries (WEB-INF/lib/elasticsearch-*.jar). Can you list all libraries included in Atlas packaging due to this change?
>> can you enlighten me as to the issue with including the jars in the default distribution?
Have you validated running Atlas with Solr after this change (which packages elastic-search JARs)? If you already validated, it should be okay to push this commit.
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
-----------------------------------------------------------
On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 16, 2018, 2:29 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/5/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when building for solr?
>
> Pierre Padovani wrote:
> Yes... however the new 5.x version of the Elasticsearch jars no longer shade 3rd party libraries so are far safer than the earlier versions. So we have a choice, include the jars, or force users who wish to use Elasticsearch as the search backend to copy in the jars manually.
>
> Madhan Neethiraj wrote:
> Pierre - I would suggest to not package libraries from multiple search backends in Atlas. It should be done either via profiles or as you suggest by manually copying necessary libraries.
>
> Pierre Padovani wrote:
> Not to be obstinate about this... but have you tried to install a different set of libraries into the default atlas distribution? The lib directory doesn't exist, so you have to:
>
> -start atlas
> -have it unpack itself
> -fail to start
> -stop atlas
> -copy in the libraries needed
> -start atlas
>
> The above assumes the user isn't going to remove the solr jars... since they likely don't know about them. In the end it is very likely that they will end up with both search backend jars if they choose to use Elasticsearch.
>
> IMHO - This indicates that there is a bigger distribution packaging issue in place. What we should do is not include any backend or search jars in the default war build. Instead include them in the zip, and provide a setup script that allows the user to choose their setup which in turn unpacks and copies in the require jars.
>
> Finally, can you enlighten me as to the issue with including the jars in the default distribution? I understand it is your preference, but I fail to grasp what the conflict happens to be except for overall package size.
>
> Madhan Neethiraj wrote:
> This removes existing exclusion of all elasicsearch libraries (WEB-INF/lib/elasticsearch-*.jar). Can you list all libraries included in Atlas packaging due to this change?
>
> >> can you enlighten me as to the issue with including the jars in the default distribution?
> Have you validated running Atlas with Solr after this change (which packages elastic-search JARs)? If you already validated, it should be okay to push this commit.
Here are the two jars that are included in the build:
elasticsearch-rest-client-5.6.4.jar
elasticsearch-rest-high-level-client-5.6.4.jar
To verify that there is no conflict, I rebuilt with embedded hbase and solr, then copied in the above two jars. I was able to run, create an entity and search for it without any issues.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
-----------------------------------------------------------
On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 16, 2018, 2:29 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/5/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when building for solr?
Yes... however the new 5.x version of the Elasticsearch jars no longer shade 3rd party libraries so are far safer than the earlier versions. So we have a choice, include the jars, or force users who wish to use Elasticsearch as the search backend to copy in the jars manually.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
-----------------------------------------------------------
On May 8, 2018, 3 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 8, 2018, 3 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/4/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when building for solr?
>
> Pierre Padovani wrote:
> Yes... however the new 5.x version of the Elasticsearch jars no longer shade 3rd party libraries so are far safer than the earlier versions. So we have a choice, include the jars, or force users who wish to use Elasticsearch as the search backend to copy in the jars manually.
>
> Madhan Neethiraj wrote:
> Pierre - I would suggest to not package libraries from multiple search backends in Atlas. It should be done either via profiles or as you suggest by manually copying necessary libraries.
Not to be obstinate about this... but have you tried to install a different set of libraries into the default atlas distribution? The lib directory doesn't exist, so you have to:
-start atlas
-have it unpack itself
-fail to start
-stop atlas
-copy in the libraries needed
-start atlas
The above assumes the user isn't going to remove the solr jars... since they likely don't know about them. In the end it is very likely that they will end up with both search backend jars if they choose to use Elasticsearch.
IMHO - This indicates that there is a bigger distribution packaging issue in place. What we should do is not include any backend or search jars in the default war build. Instead include them in the zip, and provide a setup script that allows the user to choose their setup which in turn unpacks and copies in the require jars.
Finally, can you enlighten me as to the issue with including the jars in the default distribution? I understand it is your preference, but I fail to grasp what the conflict happens to be except for overall package size.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
-----------------------------------------------------------
On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 16, 2018, 2:29 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/5/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Madhan Neethiraj <ma...@apache.org>.
> On May 7, 2018, 10:39 p.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 675 (original), 676 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017714#file2017714line676>
> >
> > Will this change end up including elasticsearch libraries even when building for solr?
>
> Pierre Padovani wrote:
> Yes... however the new 5.x version of the Elasticsearch jars no longer shade 3rd party libraries so are far safer than the earlier versions. So we have a choice, include the jars, or force users who wish to use Elasticsearch as the search backend to copy in the jars manually.
Pierre - I would suggest to not package libraries from multiple search backends in Atlas. It should be done either via profiles or as you suggest by manually copying necessary libraries.
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
-----------------------------------------------------------
On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 16, 2018, 2:29 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/5/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202599
-----------------------------------------------------------
common/src/main/java/org/apache/atlas/repository/Constants.java
Line 55 (original), 59 (patched)
<https://reviews.apache.org/r/66064/#comment284487>
given getTypePropertyKey() method replaces "." with "_" for elastic-search, continue to retain "." in the value here.
pom.xml
Line 675 (original), 676 (patched)
<https://reviews.apache.org/r/66064/#comment284488>
Will this change end up including elasticsearch libraries even when building for solr?
pom.xml
Line 694 (original), 695 (patched)
<https://reviews.apache.org/r/66064/#comment284489>
Will this change end up including elasticsearch libraries even when building for solr?
- Madhan Neethiraj
On May 7, 2018, 9:28 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 7, 2018, 9:28 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/3/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> > File "./bin/atlas_start.py", line 163, in <module>
> > File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
Fixed in the updated patch.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
-----------------------------------------------------------
On May 8, 2018, 3 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 8, 2018, 3 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/4/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> > File "./bin/atlas_start.py", line 163, in <module>
> > File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
>
> Pierre Padovani wrote:
> Fixed in the updated patch.
>
> Pierre Padovani wrote:
> Any updates on this review?
>
> David Radley wrote:
> Hi Pierre - there are outstanding issues against this entry - which normally indicates there is work for the controbuter to do. These need to be fixed or dropped before a subsequent review.
My bad, I had uploaded a new patch, but didn't click the 'fixed' buttons. I've rebased and uploaded a new version of the patch.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
-----------------------------------------------------------
On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 16, 2018, 2:29 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/5/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> > File "./bin/atlas_start.py", line 163, in <module>
> > File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
>
> Pierre Padovani wrote:
> Fixed in the updated patch.
Any updates on this review?
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
-----------------------------------------------------------
On May 8, 2018, 3 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 8, 2018, 3 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/4/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by David Radley <da...@apache.org>.
> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> > File "./bin/atlas_start.py", line 163, in <module>
> > File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
>
> Pierre Padovani wrote:
> Fixed in the updated patch.
>
> Pierre Padovani wrote:
> Any updates on this review?
Hi Pierre - there are outstanding issues against this entry - which normally indicates there is work for the controbuter to do. These need to be fixed or dropped before a subsequent review.
- David
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
-----------------------------------------------------------
On May 8, 2018, 3 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 8, 2018, 3 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/4/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
-----------------------------------------------------------
On executing ./bin/atlas_start.py displays these messages:
./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh: line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad substitution
Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
Traceback (most recent call last):
File "./bin/atlas_start.py", line 163, in <module>
File "./bin/atlas_start.py", line 92, in main
Exception: ('Could not find hbase-site.xml in %s. Please set env var HBASE_CONF_DIR to the hbase client conf dir', './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
- Ashutosh Mestry
On May 7, 2018, 9:28 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 7, 2018, 9:28 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/3/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202597
-----------------------------------------------------------
Ship it!
Ship It!
- Ashutosh Mestry
On May 7, 2018, 9:28 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 7, 2018, 9:28 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/3/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 16, 2018, 6:11 p.m., Madhan Neethiraj wrote:
> > docs/src/site/twiki/Configuration.twiki
> > Lines 47 (patched)
> > <https://reviews.apache.org/r/66064/diff/5/?file=2024037#file2024037line47>
> >
> > Per earlier discussions, support for Elasticsearch will be tech-preview for Atlas 1.0.0 release. Please update the documentation to reflect this.
> >
> > Does Elasticsearch support kerberos authentication? If yes, have you validated Atlas working with Elasticsearch in kerberized environment?
I updated the application properties to reflect the answers to the above. It does not support kerberos out of the box. There is a plugin that provides security from the company that makes Elasticsearch, and I referenced it as well as the recommended method for securing elasticsearch in the JanusGraph ducomentation.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review203265
-----------------------------------------------------------
On May 17, 2018, 2:39 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 17, 2018, 2:39 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/6/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review203265
-----------------------------------------------------------
docs/src/site/twiki/Configuration.twiki
Lines 47 (patched)
<https://reviews.apache.org/r/66064/#comment285368>
Per earlier discussions, support for Elasticsearch will be tech-preview for Atlas 1.0.0 release. Please update the documentation to reflect this.
Does Elasticsearch support kerberos authentication? If yes, have you validated Atlas working with Elasticsearch in kerberized environment?
- Madhan Neethiraj
On May 16, 2018, 2:29 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 16, 2018, 2:29 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/5/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review203349
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On May 17, 2018, 2:39 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 17, 2018, 2:39 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
> distro/pom.xml 379ff0d39
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/6/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
-----------------------------------------------------------
(Updated May 17, 2018, 2:39 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
Updated the comments in the application properties for configuring elasticsearch: Indicated tech preview, and security options
Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478
Repository: atlas
Description
-------
This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
**Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
distro/pom.xml 379ff0d39
distro/src/bin/atlas_config.py 9062da649
distro/src/bin/atlas_start.py 61d69eb21
distro/src/bin/atlas_stop.py 94c3d6d46
distro/src/conf/atlas-env.sh 053cbd500
distro/src/main/assemblies/standalone-package.xml dc88add56
docs/src/site/twiki/Configuration.twiki 63c3fce96
docs/src/site/twiki/HighAvailability.twiki 4270d0974
docs/src/site/twiki/InstallationSteps.twiki dca0618e3
graphdb/janus/pom.xml 016a09c33
pom.xml 3469a393a
webapp/pom.xml 03b84087f
Diff: https://reviews.apache.org/r/66064/diff/6/
Changes: https://reviews.apache.org/r/66064/diff/5-6/
Testing
-------
We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
Thanks,
Pierre Padovani
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
-----------------------------------------------------------
(Updated May 16, 2018, 2:29 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
Rebased against master.
Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478
Repository: atlas
Description
-------
This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
**Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 3732556ae
distro/pom.xml 379ff0d39
distro/src/bin/atlas_config.py 9062da649
distro/src/bin/atlas_start.py 61d69eb21
distro/src/bin/atlas_stop.py 94c3d6d46
distro/src/conf/atlas-env.sh 053cbd500
distro/src/main/assemblies/standalone-package.xml dc88add56
docs/src/site/twiki/Configuration.twiki 63c3fce96
docs/src/site/twiki/HighAvailability.twiki 4270d0974
docs/src/site/twiki/InstallationSteps.twiki dca0618e3
graphdb/janus/pom.xml 016a09c33
pom.xml 3469a393a
webapp/pom.xml 03b84087f
Diff: https://reviews.apache.org/r/66064/diff/5/
Changes: https://reviews.apache.org/r/66064/diff/4-5/
Testing
-------
We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
Thanks,
Pierre Padovani
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
-----------------------------------------------------------
(Updated May 8, 2018, 3 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
Fixes based on review
Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478
Repository: atlas
Description
-------
This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
**Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
distro/pom.xml 6431fd86d
distro/src/bin/atlas_config.py 9062da649
distro/src/bin/atlas_start.py 61d69eb21
distro/src/bin/atlas_stop.py 94c3d6d46
distro/src/conf/atlas-env.sh 053cbd500
distro/src/main/assemblies/standalone-package.xml dc88add56
docs/src/site/twiki/Configuration.twiki 63c3fce96
docs/src/site/twiki/HighAvailability.twiki 4270d0974
docs/src/site/twiki/InstallationSteps.twiki dca0618e3
graphdb/janus/pom.xml 016a09c33
pom.xml 3469a393a
webapp/pom.xml 03b84087f
Diff: https://reviews.apache.org/r/66064/diff/4/
Changes: https://reviews.apache.org/r/66064/diff/3-4/
Testing
-------
We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
Thanks,
Pierre Padovani
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
> On May 7, 2018, 11:35 p.m., Ashutosh Mestry wrote:
> > graphdb/janus/pom.xml
> > Lines 121 (patched)
> > <https://reviews.apache.org/r/66064/diff/3/?file=2017713#file2017713line121>
> >
> > This is getting included even when the profile is not selected.
Per the comment about elasticsearch jars in the main distribution.
The new version is safer now after Elastic removed all of the shaded jars... and smaller.
We either keep them, or force them to download and copy it into their distribution during setup/installation.
- Pierre
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202602
-----------------------------------------------------------
On May 8, 2018, 3 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 8, 2018, 3 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/4/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202602
-----------------------------------------------------------
graphdb/janus/pom.xml
Lines 121 (patched)
<https://reviews.apache.org/r/66064/#comment284490>
This is getting included even when the profile is not selected.
- Ashutosh Mestry
On May 7, 2018, 9:28 p.m., Pierre Padovani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> -----------------------------------------------------------
>
> (Updated May 7, 2018, 9:28 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> -------
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/3/
>
>
> Testing
> -------
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
-----------------------------------------------------------
(Updated May 7, 2018, 9:28 p.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
Rebased against master
Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478
Repository: atlas
Description
-------
This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
**Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
distro/pom.xml 6431fd86d
distro/src/bin/atlas_config.py 9062da649
distro/src/bin/atlas_start.py 61d69eb21
distro/src/bin/atlas_stop.py 94c3d6d46
distro/src/conf/atlas-env.sh 053cbd500
distro/src/main/assemblies/standalone-package.xml dc88add56
docs/src/site/twiki/Configuration.twiki 63c3fce96
docs/src/site/twiki/HighAvailability.twiki 4270d0974
docs/src/site/twiki/InstallationSteps.twiki dca0618e3
graphdb/janus/pom.xml 016a09c33
pom.xml 3469a393a
webapp/pom.xml 03b84087f
Diff: https://reviews.apache.org/r/66064/diff/3/
Changes: https://reviews.apache.org/r/66064/diff/2-3/
Testing
-------
We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
Thanks,
Pierre Padovani
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with
JanusGraph 0.2.0
Posted by Pierre Padovani <pi...@padovani.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/
-----------------------------------------------------------
(Updated April 13, 2018, 2:55 a.m.)
Review request for atlas, David Radley, Madhan Neethiraj, and Sarath Subramanian.
Changes
-------
This update is a rebase against master. It also resolves an inconsistent startup of elasticsearch.
This partially resolves some of the issues found in this review, but I believe further discussion is needed to resolve things correctly.
Issues:
- Berkley db is excluded from the distribution, so there is this copy hack that must happen. This has always been the case and falls outside the scope of this patch
- The way I configure the Constants is not ideal. I am in need of guidance on where/how to implement this fix as my understanding of the internals of Atlas is limited.
- The connection to elasticsearch is now done by REST and not by native client. The jars do not contain any shaded 3rd party jars, and are needed for the REST client. If we build/distribute a full distribution that isn't based on a particular profile, these jars will be required if someone wants to use Elasticsearch. They should be harmless due to the lack of shaded 3rd party jars.
- I need to add some documentation around using this profile, but wish to hold off until we have agreement on what and how to deal with the Elasticsearch jars.
To Run this profile:
mvn clean install -Pdist,berkeley-elasticsearch -DskipTests
cd distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT
./bin/atlas_start.py
./bin/atlas_stop.py <-- Atlas will NOT start correctly becuase the berkley db jar is missing, starting and stopping will create the lib dir where the berkley db jar is needed.
cp ../../../../webapp/target/atlas-webapp-1.0.0-SNAPSHOT/WEB-INF/lib/je-7.3.7.jar server/webapp/atlas/WEB-INF/lib
./bin/atlas_start.py
Bugs: ATLAS-2478
https://issues.apache.org/jira/browse/ATLAS-2478
Repository: atlas
Description
-------
This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates documentation.
Included with this patch is an update to the berkley-elasticsearch profile to automatically download and include elasticsearch as a side application much like solr is. Updates to the start/stop/conf scripts are included as well.
NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There are six constants that are incorrectly named with a '.' (dot). This is not supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS** the field names can be collectively thought of as an object. In the case of the fields defined in the Constants.java file, 'type' is defined as a string field, and 'type.name' is also defined as a string field. Elasticsearch sees this as an error, since it cannot convert type to an object. The fix included simply changes the field names from using a '.' (dot) to an '_' (underscore). This should NOT affect compatibility with hbase/solr for new installs. For existing installations, a reindex will be required as the field names will have changed.
**Query**: There is a way we can simplify integration/unit tests for the in-memory graph store by using a maven plugin that will download and run an elasticsearch node. This is nothing more than a maven change, and change to the atlas-application.properties to switch to elasticsearch from solr. I did not implement this, but am curious if this change would be desired. If so, this can be done with a separate ticket.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 01e49157d
distro/pom.xml 1f4c6d557
distro/src/bin/atlas_config.py 9062da649
distro/src/bin/atlas_start.py 61d69eb21
distro/src/bin/atlas_stop.py 94c3d6d46
distro/src/conf/atlas-env.sh 9213f488b
distro/src/main/assemblies/standalone-package.xml dc2a66ba9
docs/src/site/twiki/Configuration.twiki 63c3fce96
docs/src/site/twiki/HighAvailability.twiki 4270d0974
docs/src/site/twiki/InstallationSteps.twiki dca0618e3
graphdb/janus/pom.xml 016a09c33
pom.xml 4d2ab7391
webapp/pom.xml 284f538f7
Diff: https://reviews.apache.org/r/66064/diff/2/
Changes: https://reviews.apache.org/r/66064/diff/1-2/
Testing
-------
We currently use this fix with our Atlas setup in a fork of the Atlas code, and have found no issues with it. Additionally, with the update to the berkley-elasticsearch profile, I have extensively tested that profile to make sure that management of Elasticsearch functioned correctly.
Thanks,
Pierre Padovani