You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Shwetha GS <ss...@hortonworks.com> on 2015/09/15 08:26:56 UTC
Review Request 38393: ATLAS-58 Make hive hook reliable
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------
Review request for atlas.
Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58
Repository: atlas
Description
-------
Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook)
4. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
5. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
6. In webapp ITs, re-used the types defined
7. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
Diffs
-----
addons/hive-bridge/pom.xml 914d8c6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
client/pom.xml 2e27930
client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
client/src/test/resources/application.properties dbd6002
common/pom.xml PRE-CREATION
common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
dashboard/public/modules/details/detailsResource.js af2f249
distro/pom.xml d42d78f
distro/src/conf/application.properties bf323a7
docs/src/site/twiki/InstallationSteps.twiki 3d0351c
notification/pom.xml b036855
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
pom.xml e8d18a2
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
repository/src/test/java/org/apache/atlas/TestUtils.java de41793
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
typesystem/pom.xml e4a1383
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
typesystem/src/main/resources/application.properties f7e2774
typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
typesystem/src/test/resources/application.properties PRE-CREATION
typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
webapp/pom.xml 6085d68
webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
webapp/src/main/resources/atlas-log4j.xml 7827c1a
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
Diff: https://reviews.apache.org/r/38393/diff/
Testing
-------
UTs and ITs
Thanks,
Shwetha GS
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
> On Sept. 18, 2015, 5:29 a.m., Suma Shivaprasad wrote:
> > addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java, line 296
> > <https://reviews.apache.org/r/38393/diff/2/?file=1073500#file1073500line296>
> >
> > is sdQualifiedName supposed to be same as tableQualifiedName ?
sd is part of both table and index. For table, sdQualifiedName = tableQualifiedName. For index, sdQualifiedName = indexQualifiedName
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99486
-----------------------------------------------------------
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99486
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java (line 277)
<https://reviews.apache.org/r/38393/#comment156363>
is sdQualifiedName supposed to be same as tableQualifiedName ?
- Suma Shivaprasad
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99498
-----------------------------------------------------------
distro/src/conf/application.properties (line 52)
<https://reviews.apache.org/r/38393/#comment156385>
Pls add docs for these configs
- Suma Shivaprasad
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
> On Sept. 18, 2015, 6:24 a.m., Suma Shivaprasad wrote:
> > client/src/main/java/org/apache/atlas/AtlasClient.java, line 110
> > <https://reviews.apache.org/r/38393/diff/2/?file=1073507#file1073507line110>
> >
> > we should make rge readTimeout configurable and also add connectTimeout
read timeout is already configurable. will add connect timeout
> On Sept. 18, 2015, 6:24 a.m., Suma Shivaprasad wrote:
> > common/src/main/java/org/apache/atlas/service/Services.java, line 38
> > <https://reviews.apache.org/r/38393/diff/2/?file=1073512#file1073512line38>
> >
> > should we add a register and unregister service method here?
register/unregister is done using guice in NotificationModule
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99487
-----------------------------------------------------------
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99487
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java (line 267)
<https://reviews.apache.org/r/38393/#comment156364>
Can we make this info ? This migght be the case of delays in synchronous calls and would be good to have visibility.
client/src/main/java/org/apache/atlas/AtlasClient.java (line 109)
<https://reviews.apache.org/r/38393/#comment156374>
we should make rge readTimeout configurable and also add connectTimeout
common/src/main/java/org/apache/atlas/service/Services.java (line 38)
<https://reviews.apache.org/r/38393/#comment156375>
should we add a register and unregister service method here?
- Suma Shivaprasad
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
> On Sept. 18, 2015, 7:04 a.m., Suma Shivaprasad wrote:
> > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java, line 139
> > <https://reviews.apache.org/r/38393/diff/2/?file=1073518#file1073518line139>
> >
> > should we throw and exceptiopn instead?
some configs like kafka endpoint don't have protocol. But I need URL object to get ports - don't want to parse it myself
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99499
-----------------------------------------------------------
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99499
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 131)
<https://reviews.apache.org/r/38393/#comment156387>
should we throw and exceptiopn instead?
- Suma Shivaprasad
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99747
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java (line 647)
<https://reviews.apache.org/r/38393/#comment156726>
we should rename this function to make the intention clear
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java (line 197)
<https://reviews.apache.org/r/38393/#comment156738>
we should include the typedefinition as well in the exception?
repository/src/main/java/org/apache/atlas/services/MetadataService.java (line 83)
<https://reviews.apache.org/r/38393/#comment156739>
pls add java doc for th args to make it clear
- Suma Shivaprasad
On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 21, 2015, 8:13 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/bin/import-hive.sh 99300e8
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> dashboard/public/modules/search/searchController.js b5e6c37
> distro/pom.xml d42d78f
> distro/src/bin/atlas_start.py 7680f04
> distro/src/conf/application.properties bf323a7
> distro/src/main/assemblies/standalone-package.xml 625396a
> docs/pom.xml 4535fe0
> docs/src/site/resources/architecture.png 826df37
> docs/src/site/resources/data-types.png 3aa1904
> docs/src/site/resources/guide-class-diagram.png ca51239
> docs/src/site/resources/guide-instance-graph.png a2c8f82
> docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
> docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
> docs/src/site/resources/images/twiki/notification.png PRE-CREATION
> docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
> docs/src/site/resources/types-instance.png 6afca21
> docs/src/site/site.xml b412569
> docs/src/site/twiki/Architecture.twiki 5f10fde
> docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
> docs/src/site/twiki/Configuration.twiki e2132cf
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> docs/src/site/twiki/QuickStart.twiki d899f95
> docs/src/site/twiki/TypeSystem.twiki 78c3503
> docs/src/site/twiki/index.twiki 40d2350
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99954
-----------------------------------------------------------
Ship it!
Ship It!
- Suma Shivaprasad
On Sept. 22, 2015, 9:09 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 22, 2015, 9:09 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/bin/import-hive.sh 99300e8
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> dashboard/public/modules/search/searchController.js b5e6c37
> distro/pom.xml d42d78f
> distro/src/bin/atlas_start.py 7680f04
> distro/src/conf/application.properties bf323a7
> distro/src/main/assemblies/standalone-package.xml 625396a
> distro/src/test/python/scripts/TestMetadata.py 57f247f
> docs/pom.xml 4535fe0
> docs/src/site/resources/architecture.png 826df37
> docs/src/site/resources/data-types.png 3aa1904
> docs/src/site/resources/guide-class-diagram.png ca51239
> docs/src/site/resources/guide-instance-graph.png a2c8f82
> docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
> docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
> docs/src/site/resources/images/twiki/notification.png PRE-CREATION
> docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
> docs/src/site/resources/types-instance.png 6afca21
> docs/src/site/site.xml b412569
> docs/src/site/twiki/Architecture.twiki 5f10fde
> docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
> docs/src/site/twiki/Configuration.twiki e2132cf
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> docs/src/site/twiki/QuickStart.twiki d899f95
> docs/src/site/twiki/TypeSystem.twiki 78c3503
> docs/src/site/twiki/index.twiki 40d2350
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------
(Updated Sept. 22, 2015, 9:09 a.m.)
Review request for atlas.
Changes
-------
Addressed review comments
Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58
Repository: atlas
Description
-------
Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
Diffs (updated)
-----
addons/hive-bridge/pom.xml 914d8c6
addons/hive-bridge/src/bin/import-hive.sh 99300e8
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
client/pom.xml 2e27930
client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
client/src/test/resources/application.properties dbd6002
common/pom.xml PRE-CREATION
common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
dashboard/public/modules/details/detailsResource.js af2f249
dashboard/public/modules/search/searchController.js b5e6c37
distro/pom.xml d42d78f
distro/src/bin/atlas_start.py 7680f04
distro/src/conf/application.properties bf323a7
distro/src/main/assemblies/standalone-package.xml 625396a
distro/src/test/python/scripts/TestMetadata.py 57f247f
docs/pom.xml 4535fe0
docs/src/site/resources/architecture.png 826df37
docs/src/site/resources/data-types.png 3aa1904
docs/src/site/resources/guide-class-diagram.png ca51239
docs/src/site/resources/guide-instance-graph.png a2c8f82
docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
docs/src/site/resources/images/twiki/notification.png PRE-CREATION
docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
docs/src/site/resources/types-instance.png 6afca21
docs/src/site/site.xml b412569
docs/src/site/twiki/Architecture.twiki 5f10fde
docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
docs/src/site/twiki/Configuration.twiki e2132cf
docs/src/site/twiki/InstallationSteps.twiki 3d0351c
docs/src/site/twiki/QuickStart.twiki d899f95
docs/src/site/twiki/TypeSystem.twiki 78c3503
docs/src/site/twiki/index.twiki 40d2350
notification/pom.xml b036855
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
pom.xml e8d18a2
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
repository/src/test/java/org/apache/atlas/TestUtils.java de41793
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
typesystem/pom.xml e4a1383
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
typesystem/src/main/resources/application.properties f7e2774
typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
typesystem/src/test/resources/application.properties PRE-CREATION
typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
webapp/pom.xml 6085d68
webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
webapp/src/main/resources/atlas-log4j.xml 7827c1a
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
Diff: https://reviews.apache.org/r/38393/diff/
Testing
-------
UTs and ITs
Thanks,
Shwetha GS
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
> On Sept. 21, 2015, 3:36 p.m., Suma Shivaprasad wrote:
> > docs/src/site/twiki/TypeSystem.twiki, line 8
> > <https://reviews.apache.org/r/38393/diff/4/?file=1077987#file1077987line8>
> >
> > pls add docs for modelling unique entities by qualifiedName
I have added in EntitiesResource API. Currently, there is no documentation at all on entity creation. It will be a big effort otherwise
> On Sept. 21, 2015, 3:36 p.m., Suma Shivaprasad wrote:
> > typesystem/src/main/resources/application.properties, line 61
> > <https://reviews.apache.org/r/38393/diff/4/?file=1078031#file1078031line61>
> >
> > pls add comments for the configs
Its already in configuration twiki
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99753
-----------------------------------------------------------
On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 21, 2015, 8:13 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/bin/import-hive.sh 99300e8
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> dashboard/public/modules/search/searchController.js b5e6c37
> distro/pom.xml d42d78f
> distro/src/bin/atlas_start.py 7680f04
> distro/src/conf/application.properties bf323a7
> distro/src/main/assemblies/standalone-package.xml 625396a
> docs/pom.xml 4535fe0
> docs/src/site/resources/architecture.png 826df37
> docs/src/site/resources/data-types.png 3aa1904
> docs/src/site/resources/guide-class-diagram.png ca51239
> docs/src/site/resources/guide-instance-graph.png a2c8f82
> docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
> docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
> docs/src/site/resources/images/twiki/notification.png PRE-CREATION
> docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
> docs/src/site/resources/types-instance.png 6afca21
> docs/src/site/site.xml b412569
> docs/src/site/twiki/Architecture.twiki 5f10fde
> docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
> docs/src/site/twiki/Configuration.twiki e2132cf
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> docs/src/site/twiki/QuickStart.twiki d899f95
> docs/src/site/twiki/TypeSystem.twiki 78c3503
> docs/src/site/twiki/index.twiki 40d2350
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99753
-----------------------------------------------------------
docs/src/site/twiki/TypeSystem.twiki (line 8)
<https://reviews.apache.org/r/38393/#comment156745>
pls add docs for modelling unique entities by qualifiedName
typesystem/src/main/resources/application.properties (line 59)
<https://reviews.apache.org/r/38393/#comment156744>
pls add comments for the configs
- Suma Shivaprasad
On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 21, 2015, 8:13 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/bin/import-hive.sh 99300e8
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> dashboard/public/modules/search/searchController.js b5e6c37
> distro/pom.xml d42d78f
> distro/src/bin/atlas_start.py 7680f04
> distro/src/conf/application.properties bf323a7
> distro/src/main/assemblies/standalone-package.xml 625396a
> docs/pom.xml 4535fe0
> docs/src/site/resources/architecture.png 826df37
> docs/src/site/resources/data-types.png 3aa1904
> docs/src/site/resources/guide-class-diagram.png ca51239
> docs/src/site/resources/guide-instance-graph.png a2c8f82
> docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
> docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
> docs/src/site/resources/images/twiki/notification.png PRE-CREATION
> docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
> docs/src/site/resources/types-instance.png 6afca21
> docs/src/site/site.xml b412569
> docs/src/site/twiki/Architecture.twiki 5f10fde
> docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
> docs/src/site/twiki/Configuration.twiki e2132cf
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> docs/src/site/twiki/QuickStart.twiki d899f95
> docs/src/site/twiki/TypeSystem.twiki 78c3503
> docs/src/site/twiki/index.twiki 40d2350
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
> On Sept. 21, 2015, 3:22 p.m., Suma Shivaprasad wrote:
> > typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java, line 32
> > <https://reviews.apache.org/r/38393/diff/4/?file=1078023#file1078023line32>
> >
> > have we moved this from client to ypesystem?
yes
> On Sept. 21, 2015, 3:22 p.m., Suma Shivaprasad wrote:
> > typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java, line 121
> > <https://reviews.apache.org/r/38393/diff/4/?file=1078025#file1078025line121>
> >
> > if possible, can you pls fix the error message in other places where ValueCnversionException is being thrown curently :) Right now, we dont ge to know which attribute is causing the issue
In the validation of actual attribute, the attribute name is not available. All attribute validation are called from Struct. So, this captures everything
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99752
-----------------------------------------------------------
On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 21, 2015, 8:13 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/bin/import-hive.sh 99300e8
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> dashboard/public/modules/search/searchController.js b5e6c37
> distro/pom.xml d42d78f
> distro/src/bin/atlas_start.py 7680f04
> distro/src/conf/application.properties bf323a7
> distro/src/main/assemblies/standalone-package.xml 625396a
> docs/pom.xml 4535fe0
> docs/src/site/resources/architecture.png 826df37
> docs/src/site/resources/data-types.png 3aa1904
> docs/src/site/resources/guide-class-diagram.png ca51239
> docs/src/site/resources/guide-instance-graph.png a2c8f82
> docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
> docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
> docs/src/site/resources/images/twiki/notification.png PRE-CREATION
> docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
> docs/src/site/resources/types-instance.png 6afca21
> docs/src/site/site.xml b412569
> docs/src/site/twiki/Architecture.twiki 5f10fde
> docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
> docs/src/site/twiki/Configuration.twiki e2132cf
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> docs/src/site/twiki/QuickStart.twiki d899f95
> docs/src/site/twiki/TypeSystem.twiki 78c3503
> docs/src/site/twiki/index.twiki 40d2350
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99752
-----------------------------------------------------------
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java (line 32)
<https://reviews.apache.org/r/38393/#comment156740>
have we moved this from client to ypesystem?
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java (line 121)
<https://reviews.apache.org/r/38393/#comment156741>
if possible, can you pls fix the error message in other places where ValueCnversionException is being thrown curently :) Right now, we dont ge to know which attribute is causing the issue
- Suma Shivaprasad
On Sept. 21, 2015, 8:13 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 21, 2015, 8:13 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/bin/import-hive.sh 99300e8
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> dashboard/public/modules/search/searchController.js b5e6c37
> distro/pom.xml d42d78f
> distro/src/bin/atlas_start.py 7680f04
> distro/src/conf/application.properties bf323a7
> distro/src/main/assemblies/standalone-package.xml 625396a
> docs/pom.xml 4535fe0
> docs/src/site/resources/architecture.png 826df37
> docs/src/site/resources/data-types.png 3aa1904
> docs/src/site/resources/guide-class-diagram.png ca51239
> docs/src/site/resources/guide-instance-graph.png a2c8f82
> docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
> docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
> docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
> docs/src/site/resources/images/twiki/notification.png PRE-CREATION
> docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
> docs/src/site/resources/types-instance.png 6afca21
> docs/src/site/site.xml b412569
> docs/src/site/twiki/Architecture.twiki 5f10fde
> docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
> docs/src/site/twiki/Configuration.twiki e2132cf
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> docs/src/site/twiki/QuickStart.twiki d899f95
> docs/src/site/twiki/TypeSystem.twiki 78c3503
> docs/src/site/twiki/index.twiki 40d2350
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------
(Updated Sept. 21, 2015, 8:13 a.m.)
Review request for atlas.
Changes
-------
Fixes for import-hive.sh in end to end set-up
Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58
Repository: atlas
Description
-------
Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
Diffs (updated)
-----
addons/hive-bridge/pom.xml 914d8c6
addons/hive-bridge/src/bin/import-hive.sh 99300e8
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
client/pom.xml 2e27930
client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
client/src/test/resources/application.properties dbd6002
common/pom.xml PRE-CREATION
common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
dashboard/public/modules/details/detailsResource.js af2f249
dashboard/public/modules/search/searchController.js b5e6c37
distro/pom.xml d42d78f
distro/src/bin/atlas_start.py 7680f04
distro/src/conf/application.properties bf323a7
distro/src/main/assemblies/standalone-package.xml 625396a
docs/pom.xml 4535fe0
docs/src/site/resources/architecture.png 826df37
docs/src/site/resources/data-types.png 3aa1904
docs/src/site/resources/guide-class-diagram.png ca51239
docs/src/site/resources/guide-instance-graph.png a2c8f82
docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
docs/src/site/resources/images/twiki/notification.png PRE-CREATION
docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
docs/src/site/resources/types-instance.png 6afca21
docs/src/site/site.xml b412569
docs/src/site/twiki/Architecture.twiki 5f10fde
docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
docs/src/site/twiki/Configuration.twiki e2132cf
docs/src/site/twiki/InstallationSteps.twiki 3d0351c
docs/src/site/twiki/QuickStart.twiki d899f95
docs/src/site/twiki/TypeSystem.twiki 78c3503
docs/src/site/twiki/index.twiki 40d2350
notification/pom.xml b036855
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
pom.xml e8d18a2
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
repository/src/test/java/org/apache/atlas/TestUtils.java de41793
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
typesystem/pom.xml e4a1383
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
typesystem/src/main/resources/application.properties f7e2774
typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
typesystem/src/test/resources/application.properties PRE-CREATION
typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
webapp/pom.xml 6085d68
webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
webapp/src/main/resources/atlas-log4j.xml 7827c1a
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
Diff: https://reviews.apache.org/r/38393/diff/
Testing
-------
UTs and ITs
Thanks,
Shwetha GS
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------
(Updated Sept. 18, 2015, 11:14 a.m.)
Review request for atlas.
Changes
-------
fixed review comments. added thread pool for hive hook
Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58
Repository: atlas
Description
-------
Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
Diffs (updated)
-----
addons/hive-bridge/pom.xml 914d8c6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
client/pom.xml 2e27930
client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
client/src/test/resources/application.properties dbd6002
common/pom.xml PRE-CREATION
common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
dashboard/public/modules/details/detailsResource.js af2f249
distro/pom.xml d42d78f
distro/src/conf/application.properties bf323a7
distro/src/main/assemblies/standalone-package.xml 625396a
docs/pom.xml 4535fe0
docs/src/site/resources/architecture.png 826df37
docs/src/site/resources/data-types.png 3aa1904
docs/src/site/resources/guide-class-diagram.png ca51239
docs/src/site/resources/guide-instance-graph.png a2c8f82
docs/src/site/resources/images/twiki/architecture.png PRE-CREATION
docs/src/site/resources/images/twiki/data-types.png PRE-CREATION
docs/src/site/resources/images/twiki/guide-class-diagram.png PRE-CREATION
docs/src/site/resources/images/twiki/guide-instance-graph.png PRE-CREATION
docs/src/site/resources/images/twiki/notification.png PRE-CREATION
docs/src/site/resources/images/twiki/types-instance.png PRE-CREATION
docs/src/site/resources/types-instance.png 6afca21
docs/src/site/site.xml b412569
docs/src/site/twiki/Architecture.twiki 5f10fde
docs/src/site/twiki/Bridge-Hive.twiki PRE-CREATION
docs/src/site/twiki/Configuration.twiki e2132cf
docs/src/site/twiki/InstallationSteps.twiki 3d0351c
docs/src/site/twiki/index.twiki 40d2350
notification/pom.xml b036855
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
pom.xml e8d18a2
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
repository/src/test/java/org/apache/atlas/TestUtils.java de41793
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
typesystem/pom.xml e4a1383
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
typesystem/src/main/resources/application.properties f7e2774
typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
typesystem/src/test/resources/application.properties PRE-CREATION
typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
webapp/pom.xml 6085d68
webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
webapp/src/main/java/org/apache/atlas/web/resources/MetadataDiscoveryResource.java 9248979
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
webapp/src/main/resources/atlas-log4j.xml 7827c1a
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java 8664360
webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
Diff: https://reviews.apache.org/r/38393/diff/
Testing
-------
UTs and ITs
Thanks,
Shwetha GS
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/#review99522
-----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java (line 131)
<https://reviews.apache.org/r/38393/#comment156438>
some configs like kafka endpoint don't have protocol. But I need URL object to get ports - don't want to parse it myself
- Shwetha GS
On Sept. 15, 2015, 6:56 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38393/
> -----------------------------------------------------------
>
> (Updated Sept. 15, 2015, 6:56 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-58
> https://issues.apache.org/jira/browse/ATLAS-58
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
> Big changes:
> 1. Concept of service that are started and stopped at atlas start and stop
> 2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
> 3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
> 4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
> 5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
> 6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
> 7. In webapp ITs, re-used the types defined
> 8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
>
> Pending:
> 1. Entity updates like alter table commands are not handlded. Will create another jira for this
> 2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
>
>
> Diffs
> -----
>
> addons/hive-bridge/pom.xml 914d8c6
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
> addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
> client/pom.xml 2e27930
> client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
> client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
> client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
> client/src/test/resources/application.properties dbd6002
> common/pom.xml PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
> common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
> dashboard/public/modules/details/detailsResource.js af2f249
> distro/pom.xml d42d78f
> distro/src/conf/application.properties bf323a7
> docs/src/site/twiki/InstallationSteps.twiki 3d0351c
> notification/pom.xml b036855
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
> notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
> notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
> notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
> pom.xml e8d18a2
> repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
> repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
> repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
> repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
> repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
> repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
> repository/src/test/java/org/apache/atlas/TestUtils.java de41793
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
> repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
> repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
> repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
> repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
> repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
> repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
> repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
> repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
> repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
> repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
> repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
> repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
> typesystem/pom.xml e4a1383
> typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
> typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
> typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
> typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
> typesystem/src/main/resources/application.properties f7e2774
> typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
> typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
> typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
> typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
> typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
> typesystem/src/test/resources/application.properties PRE-CREATION
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
> typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
> webapp/pom.xml 6085d68
> webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
> webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
> webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
> webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
> webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
> webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
> webapp/src/main/resources/atlas-log4j.xml 7827c1a
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
> webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
> webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
> webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
> webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
> webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
> webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
> webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38393/diff/
>
>
> Testing
> -------
>
> UTs and ITs
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------
(Updated Sept. 15, 2015, 6:56 a.m.)
Review request for atlas.
Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58
Repository: atlas
Description (updated)
-------
Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook) - backward incompatible
4. Moved submit and list from EntityResource to EntitiesResource - backward incompatible
5. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
6. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
7. In webapp ITs, re-used the types defined
8. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
Diffs
-----
addons/hive-bridge/pom.xml 914d8c6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
client/pom.xml 2e27930
client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
client/src/test/resources/application.properties dbd6002
common/pom.xml PRE-CREATION
common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
dashboard/public/modules/details/detailsResource.js af2f249
distro/pom.xml d42d78f
distro/src/conf/application.properties bf323a7
docs/src/site/twiki/InstallationSteps.twiki 3d0351c
notification/pom.xml b036855
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
pom.xml e8d18a2
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
repository/src/test/java/org/apache/atlas/TestUtils.java de41793
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
typesystem/pom.xml e4a1383
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
typesystem/src/main/resources/application.properties f7e2774
typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
typesystem/src/test/resources/application.properties PRE-CREATION
typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
webapp/pom.xml 6085d68
webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
webapp/src/main/resources/atlas-log4j.xml 7827c1a
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
Diff: https://reviews.apache.org/r/38393/diff/
Testing
-------
UTs and ITs
Thanks,
Shwetha GS
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------
(Updated Sept. 15, 2015, 6:50 a.m.)
Review request for atlas.
Changes
-------
Fixed hive-bridge IT failures
Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58
Repository: atlas
Description
-------
Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook)
4. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
5. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
6. In webapp ITs, re-used the types defined
7. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
Diffs (updated)
-----
addons/hive-bridge/pom.xml 914d8c6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
client/pom.xml 2e27930
client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
client/src/test/resources/application.properties dbd6002
common/pom.xml PRE-CREATION
common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
dashboard/public/modules/details/detailsResource.js af2f249
distro/pom.xml d42d78f
distro/src/conf/application.properties bf323a7
docs/src/site/twiki/InstallationSteps.twiki 3d0351c
notification/pom.xml b036855
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
pom.xml e8d18a2
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
repository/src/test/java/org/apache/atlas/TestUtils.java de41793
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
typesystem/pom.xml e4a1383
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
typesystem/src/main/resources/application.properties f7e2774
typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
typesystem/src/test/resources/application.properties PRE-CREATION
typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
webapp/pom.xml 6085d68
webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
webapp/src/main/resources/atlas-log4j.xml 7827c1a
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
Diff: https://reviews.apache.org/r/38393/diff/
Testing
-------
UTs and ITs
Thanks,
Shwetha GS
Re: Review Request 38393: ATLAS-58 Make hive hook reliable
Posted by Shwetha GS <ss...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38393/
-----------------------------------------------------------
(Updated Sept. 15, 2015, 6:33 a.m.)
Review request for atlas.
Bugs: ATLAS-58
https://issues.apache.org/jira/browse/ATLAS-58
Repository: atlas
Description (updated)
-------
Hive hook sends notification messages (list of entities). The Notification consumer on server side consumes these messages and registers the entities. The server handles de-duping of entities based on the unique attribute of the entity
Big changes:
1. Concept of service that are started and stopped at atlas start and stop
2. De-duping of entities on server based on any unique attribute for the entity. If entity doesn't have any unique attribute, de-duping is not done and new entity is created
3. Changed entity submit API to take list of entities instead of just 1 entity (required for hive hook)
4. Moved security tests from integration tests to unit tests - as they were creating issues with server start as jetty already starts another server for integration tests
5. Removed some duplicate tests from repository module (the same tests exist in typesystem module as well)
6. In webapp ITs, re-used the types defined
7. Hive hook now sends notifications instead of registering entities. Sending notification is done synchronously. So, this adds to hive command execution delay. But this also makes it reliable
Pending:
1. Entity updates like alter table commands are not handlded. Will create another jira for this
2. Webapp jetty plugin doesn't shutdown embedded kafka at the end of integration tests. So, hive bridge ITs fail. Hive bridge ITs pass if run on their own. Still checking on this
Diffs
-----
addons/hive-bridge/pom.xml 914d8c6
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 4d1af4e
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java fe26446
addons/hive-bridge/src/main/java/org/apache/atlas/hive/model/HiveDataModelGenerator.java cb8359e
addons/hive-bridge/src/site/twiki/Bridge-Hive.twiki 647af27
addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/HiveHookIT.java 2bf3aba
client/pom.xml 2e27930
client/src/main/java/org/apache/atlas/ApplicationProperties.java 738ec53
client/src/main/java/org/apache/atlas/AtlasClient.java 1c7d62a
client/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java 4acb3e1
client/src/test/resources/application.properties dbd6002
common/pom.xml PRE-CREATION
common/src/main/java/org/apache/atlas/service/Service.java PRE-CREATION
common/src/main/java/org/apache/atlas/service/Services.java PRE-CREATION
dashboard/public/modules/details/detailsResource.js af2f249
distro/pom.xml d42d78f
distro/src/conf/application.properties bf323a7
docs/src/site/twiki/InstallationSteps.twiki 3d0351c
notification/pom.xml b036855
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 9978275
notification/src/main/java/org/apache/atlas/kafka/KafkaNotificationProvider.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 36a62f0
notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 0951124
notification/src/main/java/org/apache/atlas/notification/NotificationModule.java db17e35
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 02752dc
pom.xml e8d18a2
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 3d47293
repository/src/main/java/org/apache/atlas/RepositoryMetadataModule.java fbd01de
repository/src/main/java/org/apache/atlas/listener/EntityChangeListener.java f58d6de
repository/src/main/java/org/apache/atlas/repository/EntityExistsException.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/MetadataRepository.java 1b0faa5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java cc10ea3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java ceb8a65
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 56168db
repository/src/main/java/org/apache/atlas/services/MetadataService.java 8f82f88
repository/src/test/java/org/apache/atlas/RepositoryServiceLoadingTest.java 18b1d1b
repository/src/test/java/org/apache/atlas/TestUtils.java de41793
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java 127e6d7
repository/src/test/java/org/apache/atlas/discovery/HiveLineageServiceTest.java 2c7d61a
repository/src/test/java/org/apache/atlas/repository/BaseTest.java 8924c37
repository/src/test/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepositoryTest.java 487ac84
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java be99efe
repository/src/test/java/org/apache/atlas/repository/memory/ClassTest.java ed747b2
repository/src/test/java/org/apache/atlas/repository/memory/EnumTest.java 4ce1085
repository/src/test/java/org/apache/atlas/repository/memory/InstanceE2ETest.java 3c29bf8
repository/src/test/java/org/apache/atlas/repository/memory/StorageTest.java c329caf
repository/src/test/java/org/apache/atlas/repository/memory/StructTest.java 3ab7020
repository/src/test/java/org/apache/atlas/repository/memory/TraitTest.java bfdc8b1
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java PRE-CREATION
repository/src/test/scala/org/apache/atlas/query/HiveTitanSample.scala eef13df
repository/src/test/scala/org/apache/atlas/query/QueryTestsUtils.scala ec21886
typesystem/pom.xml e4a1383
typesystem/src/main/java/org/apache/atlas/ApplicationProperties.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/TypeExistsException.java PRE-CREATION
typesystem/src/main/java/org/apache/atlas/typesystem/persistence/StructInstance.java 309ab11
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeDefinition.java 31b5f47
typesystem/src/main/java/org/apache/atlas/typesystem/types/AttributeInfo.java e74f8d5
typesystem/src/main/java/org/apache/atlas/typesystem/types/ObjectGraphWalker.java 399902c
typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java d1e8dbd
typesystem/src/main/java/org/apache/atlas/typesystem/types/ValueConversionException.java 5cdee8d
typesystem/src/main/resources/application.properties f7e2774
typesystem/src/main/resources/atlas-log4j.xml PRE-CREATION
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala 7a681bf
typesystem/src/test/java/org/apache/atlas/ApplicationPropertiesTest.java PRE-CREATION
typesystem/src/test/java/org/apache/atlas/typesystem/json/SerializationJavaTest.java a8c58bd
typesystem/src/test/java/org/apache/atlas/typesystem/types/BaseTest.java 1d80681
typesystem/src/test/java/org/apache/atlas/typesystem/types/TypeSystemTest.java e8bbb0c
typesystem/src/test/resources/application.properties PRE-CREATION
typesystem/src/test/scala/org/apache/atlas/typesystem/json/SerializationTest.scala d14ac72
typesystem/src/test/scala/org/apache/atlas/typesystem/json/TypesSerializationTest.scala ba4782c
webapp/pom.xml 6085d68
webapp/src/main/java/org/apache/atlas/Main.java 7b13f3d
webapp/src/main/java/org/apache/atlas/examples/QuickStart.java b32b403
webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java bcf115f
webapp/src/main/java/org/apache/atlas/web/resources/EntitiesResource.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java dcb460d
webapp/src/main/java/org/apache/atlas/web/resources/TypesResource.java d6d5915
webapp/src/main/java/org/apache/atlas/web/service/SecureEmbeddedServer.java 1b2192c
webapp/src/main/resources/atlas-log4j.xml 7827c1a
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/listeners/TestGuiceServletConfig.java 3ba6387
webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 884d4de
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java e6a7325
webapp/src/test/java/org/apache/atlas/web/resources/HiveLineageJerseyResourceIT.java a884b5f
webapp/src/test/java/org/apache/atlas/web/security/BaseSecurityTest.java 8af4a7e
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosIT.java 0b95b7a
webapp/src/test/java/org/apache/atlas/web/security/NegativeSSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosIT.java f00ac64
webapp/src/test/java/org/apache/atlas/web/security/SSLAndKerberosTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/security/SSLIT.java 3e23185
webapp/src/test/java/org/apache/atlas/web/security/SSLTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerIT.java e1f9b54
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerITBase.java f7c3625
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTest.java PRE-CREATION
webapp/src/test/java/org/apache/atlas/web/service/SecureEmbeddedServerTestBase.java PRE-CREATION
Diff: https://reviews.apache.org/r/38393/diff/
Testing
-------
UTs and ITs
Thanks,
Shwetha GS