You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Jeff Hagelberg <jn...@us.ibm.com> on 2016/06/15 22:15:53 UTC
Review Request 48760: ATLAS-908: Force index creation up front
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48760/
-----------------------------------------------------------
Review request for atlas and David Kantor.
Bugs: ATLAS-908
https://issues.apache.org/jira/browse/ATLAS-908
Repository: atlas
Description
-------
ATLAS-908: Force index creation up front
Diffs
-----
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java c4c767827014a265471250306f35316a0b4ed67f
Diff: https://reviews.apache.org/r/48760/diff/
Testing
-------
Ran repository tests, no issues found.
Thanks,
Jeff Hagelberg
Re: Review Request 48760: ATLAS-908: Force index creation up front
Posted by David Kantor <dk...@us.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48760/#review137871
-----------------------------------------------------------
Ship it!
Ship It!
- David Kantor
On June 15, 2016, 10:35 p.m., Jeff Hagelberg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48760/
> -----------------------------------------------------------
>
> (Updated June 15, 2016, 10:35 p.m.)
>
>
> Review request for atlas and David Kantor.
>
>
> Bugs: ATLAS-908
> https://issues.apache.org/jira/browse/ATLAS-908
>
>
> Repository: atlas
>
>
> Description
> -------
>
> When starting Atlas for the first time, the graph indices do not actually get created up front when starting Atlas. This is a problem because you can configure Titan to not allow queries to run if there is not an graph index that can be used (using the for. In environments like this, the Atlas startup fails because it attempts to restore the typesystem (which runs a query) before creating the base graph indices.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java c4c767827014a265471250306f35316a0b4ed67f
>
> Diff: https://reviews.apache.org/r/48760/diff/
>
>
> Testing
> -------
>
> Ran repository tests, no issues found.
>
>
> Thanks,
>
> Jeff Hagelberg
>
>
Re: Review Request 48760: ATLAS-908: Force index creation up front
Posted by Jeff Hagelberg <jn...@us.ibm.com>.
> On June 16, 2016, 12:21 a.m., Hemanth Yamijala wrote:
> > I am unable to comment on JIRA because of issues with Apache JIRA instance.
> >
> > However, from a log of a recently launched one, I see this sequence:
> >
> > 2016-06-15 22:57:09,910 INFO - [main:] ~ Indexes do not exist, Creating indexes for titanGraph. (GraphBackedSearchIndexer:97)
> > 2016-06-15 22:57:11,205 INFO - [main:] ~ Index creation for global keys complete. (GraphBackedSearchIndexer:132)
> >
> > and then
> >
> > 2016-06-15 22:57:11,260 INFO - [main:] ~ Restoring type system from the store (DefaultMetadataService:142)
> > 2016-06-15 22:57:11,341 INFO - [main:] ~ Initializing type system for the first time. (DefaultMetadataService:145)
> >
> > If you see the timestamps, Graph indexes are indeed created before the Type system is either restored or created for first time.
> >
> > Can you please explain why you think indexes are not created upfront, and which indexes are you referring to?
>
> Jeff Hagelberg wrote:
> I'm referring to the indices that get created by GraphBackedSearchIndexer.initialize().
>
> Hmm, this is a fix for an issue we were seeing in our titan 1 fork. We were seeing that some property keys were being created implicitly (specifically __superTypes and __traitNames) and ending up being created with the wrong multiplicity (causing all sorts of things to go wrong). We also have a titan 1 environment where graph scans are disabled where we were hitting this issue. As you can see, the fix is pretty trivial, so I wanted to try to get it into the official Atlas distribution. It's possible that this issue is specific to our test environment, though. I'll investigate and see if I can pin down a use case where it happens in the latest Atlas.
>
> Hemanth Yamijala wrote:
> GraphBackedSearchIndexer.initialize() is where the first log statement that I've shown comes from. There is an __superTypeNames and __traitNames properties for which indexes are added in this method. (I don't see __superTypes though). It will be great if you can investigate it in current Atlas code and see what is happening. As of now, this does not seem an issue in master.
I can't seem to reproduce this now. Something must have changed. I'm going to close the Jira as invalid. I'll reopen it if we see this again.
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48760/#review137895
-----------------------------------------------------------
On June 15, 2016, 10:35 p.m., Jeff Hagelberg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48760/
> -----------------------------------------------------------
>
> (Updated June 15, 2016, 10:35 p.m.)
>
>
> Review request for atlas and David Kantor.
>
>
> Bugs: ATLAS-908
> https://issues.apache.org/jira/browse/ATLAS-908
>
>
> Repository: atlas
>
>
> Description
> -------
>
> When starting Atlas for the first time, the graph indices do not actually get created up front when starting Atlas. This is a problem because you can configure Titan to not allow queries to run if there is not an graph index that can be used (using the for. In environments like this, the Atlas startup fails because it attempts to restore the typesystem (which runs a query) before creating the base graph indices.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java c4c767827014a265471250306f35316a0b4ed67f
>
> Diff: https://reviews.apache.org/r/48760/diff/
>
>
> Testing
> -------
>
> Ran repository tests, no issues found.
>
>
> Thanks,
>
> Jeff Hagelberg
>
>
Re: Review Request 48760: ATLAS-908: Force index creation up front
Posted by Hemanth Yamijala <yh...@gmail.com>.
> On June 16, 2016, 12:21 a.m., Hemanth Yamijala wrote:
> > I am unable to comment on JIRA because of issues with Apache JIRA instance.
> >
> > However, from a log of a recently launched one, I see this sequence:
> >
> > 2016-06-15 22:57:09,910 INFO - [main:] ~ Indexes do not exist, Creating indexes for titanGraph. (GraphBackedSearchIndexer:97)
> > 2016-06-15 22:57:11,205 INFO - [main:] ~ Index creation for global keys complete. (GraphBackedSearchIndexer:132)
> >
> > and then
> >
> > 2016-06-15 22:57:11,260 INFO - [main:] ~ Restoring type system from the store (DefaultMetadataService:142)
> > 2016-06-15 22:57:11,341 INFO - [main:] ~ Initializing type system for the first time. (DefaultMetadataService:145)
> >
> > If you see the timestamps, Graph indexes are indeed created before the Type system is either restored or created for first time.
> >
> > Can you please explain why you think indexes are not created upfront, and which indexes are you referring to?
>
> Jeff Hagelberg wrote:
> I'm referring to the indices that get created by GraphBackedSearchIndexer.initialize().
>
> Hmm, this is a fix for an issue we were seeing in our titan 1 fork. We were seeing that some property keys were being created implicitly (specifically __superTypes and __traitNames) and ending up being created with the wrong multiplicity (causing all sorts of things to go wrong). We also have a titan 1 environment where graph scans are disabled where we were hitting this issue. As you can see, the fix is pretty trivial, so I wanted to try to get it into the official Atlas distribution. It's possible that this issue is specific to our test environment, though. I'll investigate and see if I can pin down a use case where it happens in the latest Atlas.
GraphBackedSearchIndexer.initialize() is where the first log statement that I've shown comes from. There is an __superTypeNames and __traitNames properties for which indexes are added in this method. (I don't see __superTypes though). It will be great if you can investigate it in current Atlas code and see what is happening. As of now, this does not seem an issue in master.
- Hemanth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48760/#review137895
-----------------------------------------------------------
On June 15, 2016, 10:35 p.m., Jeff Hagelberg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48760/
> -----------------------------------------------------------
>
> (Updated June 15, 2016, 10:35 p.m.)
>
>
> Review request for atlas and David Kantor.
>
>
> Bugs: ATLAS-908
> https://issues.apache.org/jira/browse/ATLAS-908
>
>
> Repository: atlas
>
>
> Description
> -------
>
> When starting Atlas for the first time, the graph indices do not actually get created up front when starting Atlas. This is a problem because you can configure Titan to not allow queries to run if there is not an graph index that can be used (using the for. In environments like this, the Atlas startup fails because it attempts to restore the typesystem (which runs a query) before creating the base graph indices.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java c4c767827014a265471250306f35316a0b4ed67f
>
> Diff: https://reviews.apache.org/r/48760/diff/
>
>
> Testing
> -------
>
> Ran repository tests, no issues found.
>
>
> Thanks,
>
> Jeff Hagelberg
>
>
Re: Review Request 48760: ATLAS-908: Force index creation up front
Posted by Jeff Hagelberg <jn...@us.ibm.com>.
> On June 16, 2016, 12:21 a.m., Hemanth Yamijala wrote:
> > I am unable to comment on JIRA because of issues with Apache JIRA instance.
> >
> > However, from a log of a recently launched one, I see this sequence:
> >
> > 2016-06-15 22:57:09,910 INFO - [main:] ~ Indexes do not exist, Creating indexes for titanGraph. (GraphBackedSearchIndexer:97)
> > 2016-06-15 22:57:11,205 INFO - [main:] ~ Index creation for global keys complete. (GraphBackedSearchIndexer:132)
> >
> > and then
> >
> > 2016-06-15 22:57:11,260 INFO - [main:] ~ Restoring type system from the store (DefaultMetadataService:142)
> > 2016-06-15 22:57:11,341 INFO - [main:] ~ Initializing type system for the first time. (DefaultMetadataService:145)
> >
> > If you see the timestamps, Graph indexes are indeed created before the Type system is either restored or created for first time.
> >
> > Can you please explain why you think indexes are not created upfront, and which indexes are you referring to?
I'm referring to the indices that get created by GraphBackedSearchIndexer.initialize().
Hmm, this is a fix for an issue we were seeing in our titan 1 fork. We were seeing that some property keys were being created implicitly (specifically __superTypes and __traitNames) and ending up being created with the wrong multiplicity (causing all sorts of things to go wrong). We also have a titan 1 environment where graph scans are disabled where we were hitting this issue. As you can see, the fix is pretty trivial, so I wanted to try to get it into the official Atlas distribution. It's possible that this issue is specific to our test environment, though. I'll investigate and see if I can pin down a use case where it happens in the latest Atlas.
- Jeff
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48760/#review137895
-----------------------------------------------------------
On June 15, 2016, 10:35 p.m., Jeff Hagelberg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48760/
> -----------------------------------------------------------
>
> (Updated June 15, 2016, 10:35 p.m.)
>
>
> Review request for atlas and David Kantor.
>
>
> Bugs: ATLAS-908
> https://issues.apache.org/jira/browse/ATLAS-908
>
>
> Repository: atlas
>
>
> Description
> -------
>
> When starting Atlas for the first time, the graph indices do not actually get created up front when starting Atlas. This is a problem because you can configure Titan to not allow queries to run if there is not an graph index that can be used (using the for. In environments like this, the Atlas startup fails because it attempts to restore the typesystem (which runs a query) before creating the base graph indices.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java c4c767827014a265471250306f35316a0b4ed67f
>
> Diff: https://reviews.apache.org/r/48760/diff/
>
>
> Testing
> -------
>
> Ran repository tests, no issues found.
>
>
> Thanks,
>
> Jeff Hagelberg
>
>
Re: Review Request 48760: ATLAS-908: Force index creation up front
Posted by Hemanth Yamijala <yh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48760/#review137895
-----------------------------------------------------------
I am unable to comment on JIRA because of issues with Apache JIRA instance.
However, from a log of a recently launched one, I see this sequence:
2016-06-15 22:57:09,910 INFO - [main:] ~ Indexes do not exist, Creating indexes for titanGraph. (GraphBackedSearchIndexer:97)
2016-06-15 22:57:11,205 INFO - [main:] ~ Index creation for global keys complete. (GraphBackedSearchIndexer:132)
and then
2016-06-15 22:57:11,260 INFO - [main:] ~ Restoring type system from the store (DefaultMetadataService:142)
2016-06-15 22:57:11,341 INFO - [main:] ~ Initializing type system for the first time. (DefaultMetadataService:145)
If you see the timestamps, Graph indexes are indeed created before the Type system is either restored or created for first time.
Can you please explain why you think indexes are not created upfront, and which indexes are you referring to?
- Hemanth Yamijala
On June 15, 2016, 10:35 p.m., Jeff Hagelberg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48760/
> -----------------------------------------------------------
>
> (Updated June 15, 2016, 10:35 p.m.)
>
>
> Review request for atlas and David Kantor.
>
>
> Bugs: ATLAS-908
> https://issues.apache.org/jira/browse/ATLAS-908
>
>
> Repository: atlas
>
>
> Description
> -------
>
> When starting Atlas for the first time, the graph indices do not actually get created up front when starting Atlas. This is a problem because you can configure Titan to not allow queries to run if there is not an graph index that can be used (using the for. In environments like this, the Atlas startup fails because it attempts to restore the typesystem (which runs a query) before creating the base graph indices.
>
>
> Diffs
> -----
>
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java c4c767827014a265471250306f35316a0b4ed67f
>
> Diff: https://reviews.apache.org/r/48760/diff/
>
>
> Testing
> -------
>
> Ran repository tests, no issues found.
>
>
> Thanks,
>
> Jeff Hagelberg
>
>
Re: Review Request 48760: ATLAS-908: Force index creation up front
Posted by Jeff Hagelberg <jn...@us.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48760/
-----------------------------------------------------------
(Updated June 15, 2016, 10:35 p.m.)
Review request for atlas and David Kantor.
Bugs: ATLAS-908
https://issues.apache.org/jira/browse/ATLAS-908
Repository: atlas
Description (updated)
-------
When starting Atlas for the first time, the graph indices do not actually get created up front when starting Atlas. This is a problem because you can configure Titan to not allow queries to run if there is not an graph index that can be used (using the for. In environments like this, the Atlas startup fails because it attempts to restore the typesystem (which runs a query) before creating the base graph indices.
Diffs
-----
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java c4c767827014a265471250306f35316a0b4ed67f
Diff: https://reviews.apache.org/r/48760/diff/
Testing
-------
Ran repository tests, no issues found.
Thanks,
Jeff Hagelberg