You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Sarath Subramanian <sa...@apache.org> on 2019/07/09 23:53:28 UTC
Review Request 71042: ATLAS-3321: Introduce atlas metadata namespace
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
Bugs: ATLAS-3321
https://issues.apache.org/jira/browse/ATLAS-3321
Repository: atlas
Description
-------
Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
cluster name is also used to construct qualifiedName attribute in atlas -
hive_db - db_name@cluster_name
hive_table - db_name.table_name@cluster_name
hive_column - db_name.table_name.column_name@cluster_name
Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
Diffs
-----
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
Diff: https://reviews.apache.org/r/71042/diff/1/
Testing
-------
Testing in progress
Thanks,
Sarath Subramanian
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216541
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On July 12, 2019, 1:30 a.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 12, 2019, 1:30 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/bin/import-hbase.sh b9c1a4bc0
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/bin/import-hive.sh 38ec3be24
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java b3663da34
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/bin/import-kafka.sh 4bfb8b96b
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6
>
>
> Diff: https://reviews.apache.org/r/71042/diff/3/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/
-----------------------------------------------------------
(Updated July 11, 2019, 6:30 p.m.)
Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
Bugs: ATLAS-3321
https://issues.apache.org/jira/browse/ATLAS-3321
Repository: atlas
Description
-------
Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
cluster name is also used to construct qualifiedName attribute in atlas -
hive_db - db_name@cluster_name
hive_table - db_name.table_name@cluster_name
hive_column - db_name.table_name.column_name@cluster_name
Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
Diffs (updated)
-----
addons/hbase-bridge/src/bin/import-hbase.sh b9c1a4bc0
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
addons/hive-bridge/src/bin/import-hive.sh 38ec3be24
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java b3663da34
addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
addons/kafka-bridge/src/bin/import-kafka.sh 4bfb8b96b
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6
Diff: https://reviews.apache.org/r/71042/diff/3/
Changes: https://reviews.apache.org/r/71042/diff/2-3/
Testing
-------
Testing in progress
Thanks,
Sarath Subramanian
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sridhar K <Sr...@gmail.com>.
> On July 10, 2019, 10:12 p.m., Madhan Neethiraj wrote:
> > Ship It!
I am not convinved that this is correct approach.
- Sridhar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216503
-----------------------------------------------------------
On July 10, 2019, 8:11 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 10, 2019, 8:11 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java b3663da34
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6
>
>
> Diff: https://reviews.apache.org/r/71042/diff/2/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216503
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On July 10, 2019, 8:11 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 10, 2019, 8:11 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java b3663da34
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6
>
>
> Diff: https://reviews.apache.org/r/71042/diff/2/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/
-----------------------------------------------------------
(Updated July 10, 2019, 1:11 p.m.)
Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
Changes
-------
addressed review comments.
Bugs: ATLAS-3321
https://issues.apache.org/jira/browse/ATLAS-3321
Repository: atlas
Description
-------
Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
cluster name is also used to construct qualifiedName attribute in atlas -
hive_db - db_name@cluster_name
hive_table - db_name.table_name@cluster_name
hive_column - db_name.table_name.column_name@cluster_name
Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
Diffs (updated)
-----
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java b3663da34
addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6
Diff: https://reviews.apache.org/r/71042/diff/2/
Changes: https://reviews.apache.org/r/71042/diff/1-2/
Testing
-------
Testing in progress
Thanks,
Sarath Subramanian
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sridhar K <Sr...@gmail.com>.
> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I don't see a need for configuration "atlas.metadata.namespace". I suggest that we just use the <clustername>.<servicename> format. It is hard to manage with configurations like this. What would happen if this configuration is changed? What will you do with existing objects when this value is changed. This needs some kind of design exercise. Can we please have a . meeting for it.
>
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that access (CRUD) a resource run in the same cluster. There are cases where services running in a cluster that is different from where the resource resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services running in both clusters can access Hive metastore running in "cl1". In such scenarios holding on to 'clusterName' would be confusing; hence the move to a generic name 'metadata.namespace'. It makes it intutive to ask the services set this configuration, instead of confusing them with 'clusterName'. Hope this helps.
I see your point. But, the issue is in the possibility of wrongly configured names and flexibility of changing the names. In the example you gave, I would expect HMS to have the convention of "cl1.hms1" as auto generated configuration and Hive-Server resources like hive_processes should have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I think that this gives a general understanding on namespace hierarchy. If it is user configurable, user might accidentally give the same name for every cluster resulting in mixups. So, I think that we need to give some thinking. You might have given some thinking on it. If so, I would like to understand it.
- Sridhar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
-----------------------------------------------------------
On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 11:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Madhan Neethiraj <ma...@apache.org>.
> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I don't see a need for configuration "atlas.metadata.namespace". I suggest that we just use the <clustername>.<servicename> format. It is hard to manage with configurations like this. What would happen if this configuration is changed? What will you do with existing objects when this value is changed. This needs some kind of design exercise. Can we please have a . meeting for it.
Sridhar - current use of 'clusterName' works when all services that access (CRUD) a resource run in the same cluster. There are cases where services running in a cluster that is different from where the resource resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services running in both clusters can access Hive metastore running in "cl1". In such scenarios holding on to 'clusterName' would be confusing; hence the move to a generic name 'metadata.namespace'. It makes it intutive to ask the services set this configuration, instead of confusing them with 'clusterName'. Hope this helps.
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
-----------------------------------------------------------
On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 11:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sridhar K <Sr...@gmail.com>.
> On July 10, 2019, 5:55 a.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I don't see a need for configuration "atlas.metadata.namespace". I suggest that we just use the <clustername>.<servicename> format. It is hard to manage with configurations like this. What would happen if this configuration is changed? What will you do with existing objects when this value is changed. This needs some kind of design exercise. Can we please have a . meeting for it.
>
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that access (CRUD) a resource run in the same cluster. There are cases where services running in a cluster that is different from where the resource resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services running in both clusters can access Hive metastore running in "cl1". In such scenarios holding on to 'clusterName' would be confusing; hence the move to a generic name 'metadata.namespace'. It makes it intutive to ask the services set this configuration, instead of confusing them with 'clusterName'. Hope this helps.
>
> Sridhar K wrote:
> I see your point. But, the issue is in the possibility of wrongly configured names and flexibility of changing the names. In the example you gave, I would expect HMS to have the convention of "cl1.hms1" as auto generated configuration and Hive-Server resources like hive_processes should have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I think that this gives a general understanding on namespace hierarchy. If it is user configurable, user might accidentally give the same name for every cluster resulting in mixups. So, I think that we need to give some thinking. You might have given some thinking on it. If so, I would like to understand it.
If we allow the user to give values for namespace, we might have to have this information captured as first class Atlas Entity as well--for auditing, goveranance point of view.
- Sridhar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
-----------------------------------------------------------
On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 11:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sarath Subramanian <sa...@apache.org>.
> On July 9, 2019, 10:55 p.m., Sridhar K wrote:
> > Hello Sarath,
> > Is this some thing new that we are doing? I understand the reasoning. I don't see a need for configuration "atlas.metadata.namespace". I suggest that we just use the <clustername>.<servicename> format. It is hard to manage with configurations like this. What would happen if this configuration is changed? What will you do with existing objects when this value is changed. This needs some kind of design exercise. Can we please have a . meeting for it.
>
> Madhan Neethiraj wrote:
> Sridhar - current use of 'clusterName' works when all services that access (CRUD) a resource run in the same cluster. There are cases where services running in a cluster that is different from where the resource resides. For example, consider 2 clusters cl1 and cl2. HiveServer2 services running in both clusters can access Hive metastore running in "cl1". In such scenarios holding on to 'clusterName' would be confusing; hence the move to a generic name 'metadata.namespace'. It makes it intutive to ask the services set this configuration, instead of confusing them with 'clusterName'. Hope this helps.
>
> Sridhar K wrote:
> I see your point. But, the issue is in the possibility of wrongly configured names and flexibility of changing the names. In the example you gave, I would expect HMS to have the convention of "cl1.hms1" as auto generated configuration and Hive-Server resources like hive_processes should have their name spaces as "cl1.hive-server-1" and " cl2.hive-server-1". I think that this gives a general understanding on namespace hierarchy. If it is user configurable, user might accidentally give the same name for every cluster resulting in mixups. So, I think that we need to give some thinking. You might have given some thinking on it. If so, I would like to understand it.
>
> Sridhar K wrote:
> If we allow the user to give values for namespace, we might have to have this information captured as first class Atlas Entity as well--for auditing, goveranance point of view.
Sridhar,
metadata_namespace will eventually replace cluster name and going forward atlas will be using metadata namespace instead of cluster name to construct qualified name. This patch is more of replacing cluster name to metadata namespace name.
- Sarath
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
-----------------------------------------------------------
On July 9, 2019, 4:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 4:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sridhar K <Sr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216470
-----------------------------------------------------------
Hello Sarath,
Is this some thing new that we are doing? I understand the reasoning. I don't see a need for configuration "atlas.metadata.namespace". I suggest that we just use the <clustername>.<servicename> format. It is hard to manage with configurations like this. What would happen if this configuration is changed? What will you do with existing objects when this value is changed. This needs some kind of design exercise. Can we please have a . meeting for it.
- Sridhar K
On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 11:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sarath Subramanian <sa...@apache.org>.
> On July 9, 2019, 5:11 p.m., Madhan Neethiraj wrote:
> > addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
> > Line 59 (original), 60 (patched)
> > <https://reviews.apache.org/r/71042/diff/1/?file=2154121#file2154121line60>
> >
> > Configurations "atlas.cluster.name", "atlas.metadata.namespace" and their default values - are better better handled in base class AtlasHook, to avoid having to deal with this in every hook. Please review and consider refactoring.
moved the common properties to base class in AtlasHook
- Sarath
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216460
-----------------------------------------------------------
On July 10, 2019, 1:11 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 10, 2019, 1:11 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java b3663da34
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
> notification/src/main/java/org/apache/atlas/hook/AtlasHook.java 0030276c6
>
>
> Diff: https://reviews.apache.org/r/71042/diff/2/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216460
-----------------------------------------------------------
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 59 (original), 60 (patched)
<https://reviews.apache.org/r/71042/#comment303699>
Configurations "atlas.cluster.name", "atlas.metadata.namespace" and their default values - are better better handled in base class AtlasHook, to avoid having to deal with this in every hook. Please review and consider refactoring.
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 60 (original), 61 (patched)
<https://reviews.apache.org/r/71042/#comment303697>
I suggest to use earlier default value - "primary".
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
Line 109 (original), 112 (patched)
<https://reviews.apache.org/r/71042/#comment303698>
Is it necessary to retain 'clusterName' field?
Please review clusterName in other classes as well, like:
- HBaseBridge
- HiveHook
- ImpalaLineageHook
- Madhan Neethiraj
On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 11:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Sarath Subramanian <sa...@apache.org>.
> On July 9, 2019, 9:45 p.m., Ashutosh Mestry wrote:
> > addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
> > Lines 176 (patched)
> > <https://reviews.apache.org/r/71042/diff/1/?file=2154135#file2154135line176>
> >
> > Duplicate code.
All the bridge code - HiveMetastoreBridge, HBaseBridge and KafkaBridge do not share a common base class to move the method. Hence dupliccated.
- Sarath
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216468
-----------------------------------------------------------
On July 9, 2019, 4:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 4:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216468
-----------------------------------------------------------
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
Line 256 (original), 260 (patched)
<https://reviews.apache.org/r/71042/#comment303706>
Duplicate code.
addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java
Line 146 (original), 151 (patched)
<https://reviews.apache.org/r/71042/#comment303707>
Duplicate code.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Line 65 (original), 65 (patched)
<https://reviews.apache.org/r/71042/#comment303709>
Can be renamed to be made consistent with other hooks.
addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java
Lines 176 (patched)
<https://reviews.apache.org/r/71042/#comment303708>
Duplicate code.
- Ashutosh Mestry
On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 11:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>
Re: Review Request 71042: ATLAS-3321: Introduce atlas metadata
namespace
Posted by Ashutosh Mestry via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71042/#review216469
-----------------------------------------------------------
Will make sense to write a unit test that exercises the logic for getMetadataNamespace.
- Ashutosh Mestry
On July 9, 2019, 11:53 p.m., Sarath Subramanian wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71042/
> -----------------------------------------------------------
>
> (Updated July 9, 2019, 11:53 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, Madhan Neethiraj, and Nixon Rodrigues.
>
>
> Bugs: ATLAS-3321
> https://issues.apache.org/jira/browse/ATLAS-3321
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Atlas has the notion of clusterName which is used as a global namespace in which you can define the context in which entity instances are defined and grouped. A cluster name defines a scope of operation in atlas and is specified in parent entity types - hive_db (Hive), hbase_namespace (HBase), fs_path (File System), storm_topology (Storm), spark_db (Spark)
>
> cluster name is also used to construct qualifiedName attribute in atlas -
>
> hive_db - db_name@cluster_name
> hive_table - db_name.table_name@cluster_name
> hive_column - db_name.table_name.column_name@cluster_name
> Cluster name works for simple environments wherein we have a single cluster and single service instances running (single Hive or HBase service). In complex deployments - a single cluster may host multiple instances of the same service (two Hive services - Hive-1 and Hive-2), for such cases cluster name doesn't help to provide context to individual service instances and metadata groupings.
>
> So we introduce - "metadata namespace" where a single cluster can contain multiple metadata namespaces. The qualified name for atlas will be generated using metadata namespace name instead of cluster name.
>
> For each hook enabled we specify the property - "atlas.metadata.namespace" which defines the scope/context of the metadata.
>
>
> Diffs
> -----
>
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java 1825cd290
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java 1765c1848
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java 5f8f846ae
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java 76d6fe612
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java ffa56cea0
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java 05d79df29
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/HiveITBase.java cbee7bf14
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridgeTest.java d55aa5370
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/AtlasImpalaHookContext.java 1305f65a6
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/ImpalaLineageHook.java b5fdb6dbd
> addons/impala-bridge/src/main/java/org/apache/atlas/impala/hook/events/BaseImpalaEvent.java 4ea484f52
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageITBase.java f1d02375c
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/ImpalaLineageToolIT.java 8ebb385cf
> addons/impala-bridge/src/test/java/org/apache/atlas/impala/hook/ImpalaLineageHookIT.java a7b9b0cd0
> addons/kafka-bridge/src/main/java/org/apache/atlas/kafka/bridge/KafkaBridge.java 8755c9e60
> addons/sqoop-bridge/src/main/java/org/apache/atlas/sqoop/hook/SqoopHook.java 5397a4b0b
> addons/storm-bridge/src/main/java/org/apache/atlas/storm/hook/StormAtlasHook.java 97668a3c6
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b9f4118d
>
>
> Diff: https://reviews.apache.org/r/71042/diff/1/
>
>
> Testing
> -------
>
> Testing in progress
>
>
> Thanks,
>
> Sarath Subramanian
>
>