You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Sidharth Mishra <si...@apache.org> on 2021/08/04 20:33:47 UTC
[VOTE] Release Apache Atlas version 2.2.0 - rc0
Atlas team,
Apache Atlas 2.2.0 release candidate #0 is now available for vote
within the dev community. Links to the release artifacts are given
below. Please review and vote.
The vote will be open for at least 72 hours or until necessary votes
are reached.
[ ] +1 Approve
[ ] +0 No opinion
[ ] -1 Disapprove (and reason why)
Thanks,
Sid
List of all issues addressed in this release:
https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
Git tag for the release: https://github.com/apache/atlas/tree/release-2.2.0-rc0
Sources for the release:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz
Source release verification:
PGP Signature:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.asc
SHA512 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.sha512
MD5 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.md5
Keys to verify the signature of the release artifacts are available
at: https://dist.apache.org/repos/dist/release/atlas/KEYS
New features/enhancements:
- Introduced Deferred Action - Tag propagation will be handled as
a background process (AtlasTask)
- Added Re-indexing as JAVA_PATCH
- Created JavaPatch to add new super types to existing entities
- Supported for Business Metadata in Atlas Export API
- Supported HA for admin/task API
- Enhanced export service to export Terms
- Added new models and addition attributes to Atlas for Apache
Flink, GCP and ADLS Gen 2
- Added support to Configure HiveServer2 Hook to send Lineage-only Messages
- Improved and enhanced Bulk Glossary Import supporting import
with relations
- Dependency between Atlas and Kafka Zookeeper removed
- Improved performance of GraphHelper's guid and status getter methods
- Added new privilege - "admin-audits" for Atlas Admin Audits authorization
- Basic search Improvements - search text with special characters,
optimize pagination
- Updated free-text search processor to support Elasticsearch
- Refactored and enhanced Atlas Python Client supporting Python 2.7
- DSL Search enhancements - Support glossary terms and
relationship, can handle null attribute values, Java Tinkerpop
Traversals instead of GremlinScriptEngine, support search by
classification and its attribute, Caching mechanism for translated dsl
queries
- Enhanced Export/Import Service to conditionally Support
Simultaneous Operations
- Enhanced NotificationHookConsumer for Large Message Processing
- Enhanced Atlas authorization for Add/Update/Remove
classification on entities
- Provided option to add mandatory attribute to existing entity definition
- Fixed Atlas UI to load faster in case of large number of
classifications & entities
- Updated versions of dependent libraries/components: JanusGraph,
elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
gremlin, Solr, groovy, netty, Kafka
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc0
Posted by Ashutosh Mestry <am...@cloudera.com.INVALID>.
Hi Sid
+1 for Apache Atlas v2.2.0 RC0.
Here’s my verification:
* Clean build embedded-hbase-solr profile.
* Setup Atlas server in embedded HBase and Solr.
* Verified:
* Import via REST APIs.
* DSL Query performance improvement.
* AtlasTasks by importing data with long lineage and then verifying classification propagation.
* Basic search.
* Verified statistics page.
I have few more scenarios to run.
Best regards,
~ ashutosh
From: Sidharth Mishra <si...@apache.org>
Date: Wednesday, August 4, 2021 at 1:34 PM
To: dev@atlas.apache.org <de...@atlas.apache.org>
Subject: [VOTE] Release Apache Atlas version 2.2.0 - rc0
Atlas team,
Apache Atlas 2.2.0 release candidate #0 is now available for vote
within the dev community. Links to the release artifacts are given
below. Please review and vote.
The vote will be open for at least 72 hours or until necessary votes
are reached.
[ ] +1 Approve
[ ] +0 No opinion
[ ] -1 Disapprove (and reason why)
Thanks,
Sid
List of all issues addressed in this release:
https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
Git tag for the release: https://github.com/apache/atlas/tree/release-2.2.0-rc0
Sources for the release:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz
Source release verification:
PGP Signature:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.asc
SHA512 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.sha512
MD5 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.md5
Keys to verify the signature of the release artifacts are available
at: https://dist.apache.org/repos/dist/release/atlas/KEYS
New features/enhancements:
- Introduced Deferred Action - Tag propagation will be handled as
a background process (AtlasTask)
- Added Re-indexing as JAVA_PATCH
- Created JavaPatch to add new super types to existing entities
- Supported for Business Metadata in Atlas Export API
- Supported HA for admin/task API
- Enhanced export service to export Terms
- Added new models and addition attributes to Atlas for Apache
Flink, GCP and ADLS Gen 2
- Added support to Configure HiveServer2 Hook to send Lineage-only Messages
- Improved and enhanced Bulk Glossary Import supporting import
with relations
- Dependency between Atlas and Kafka Zookeeper removed
- Improved performance of GraphHelper's guid and status getter methods
- Added new privilege - "admin-audits" for Atlas Admin Audits authorization
- Basic search Improvements - search text with special characters,
optimize pagination
- Updated free-text search processor to support Elasticsearch
- Refactored and enhanced Atlas Python Client supporting Python 2.7
- DSL Search enhancements - Support glossary terms and
relationship, can handle null attribute values, Java Tinkerpop
Traversals instead of GremlinScriptEngine, support search by
classification and its attribute, Caching mechanism for translated dsl
queries
- Enhanced Export/Import Service to conditionally Support
Simultaneous Operations
- Enhanced NotificationHookConsumer for Large Message Processing
- Enhanced Atlas authorization for Add/Update/Remove
classification on entities
- Provided option to add mandatory attribute to existing entity definition
- Fixed Atlas UI to load faster in case of large number of
classifications & entities
- Updated versions of dependent libraries/components: JanusGraph,
elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
gremlin, Solr, groovy, netty, Kafka
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc0
Posted by Sidharth Mishra <si...@apache.org>.
Thanks Madhan.
I have created two JIRAs to address these two issues. The fix will be
part of rc1.
https://issues.apache.org/jira/browse/ATLAS-4390
https://issues.apache.org/jira/browse/ATLAS-4391
Regards,
Sid
On Thu, Aug 5, 2021 at 1:45 AM Madhan Neethiraj <ma...@apache.org> wrote:
>
> Sid,
>
> Validated following on Atlas 2.2.0-rc0 - looks good overall! Found couple of issues given below; please review.
>
> Thanks,
> Madhan
>
> 1. successfully built Atlas using docker, for profile external-hbase-solr (instructions in dev-support/atlas-docker/README.txt)
> 2. successfully brought up Atlas in docker containers
> 3. successfully ran quick_start
> 4. search: performed basic search, advanced search, quick search
> 5. classification: performed add, associate, propagate, update, disassociate, delete
> 6. business-metadata: performed import, add/update/remove on entities, search
> 7. glossary: performed import, add/remove on entities, searched
> 8. verified basic operations on HBase hook (create table, drop table)
> 9. verified basic operations on Hive hook (create database/table/view, lineage, alter table, drop table/database)
> 10. verified entity audit entries for above operations
>
> Issues found:
> 1. dev-support/atlas-docker/.env has BRANCH=branch-2.0; this should be updated to release-2.2.0-rc0, so that docker build would use Atlas sources from this tag
> 2. Atlas startup logs NoSuchFileException for ${atlas.log.dir}/atlas-metrics.out, which looks like incorrect metrics configuration in hadoop-metrics2.properties.
>
> 2021-08-05 07:54:07,353 WARN - [main:] ~ Error creating sink 'atlas-debug-metrics-context' (MetricsSystemImpl:508)
> org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin: org.apache.hadoop.metrics2.sink.FileSink
> at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:210)
> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:531)
> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:503)
> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:479)
> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
> at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:62)
> at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:58)
> at org.apache.atlas.web.service.DebugMetricsWrapper.init(DebugMetricsWrapper.java:43)
> ...
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:110)
> at org.apache.atlas.Atlas.main(Atlas.java:133)
> Caused by: org.apache.hadoop.metrics2.MetricsException: Error creating ${atlas.log.dir}/atlas-metrics.out
> at org.apache.hadoop.metrics2.sink.FileSink.init(FileSink.java:53)
> at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:207)
> ... 63 more
> Caused by: java.nio.file.NoSuchFileException: ${atlas.log.dir}/atlas-metrics.out
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
> at java.nio.file.Files.newOutputStream(Files.java:216)
> at org.apache.hadoop.metrics2.sink.FileSink.init(FileSink.java:50)
> ... 64 more
>
>
>
>
> On 8/4/21, 1:34 PM, "Sidharth Mishra" <si...@apache.org> wrote:
>
> Atlas team,
>
>
> Apache Atlas 2.2.0 release candidate #0 is now available for vote
> within the dev community. Links to the release artifacts are given
> below. Please review and vote.
>
>
> The vote will be open for at least 72 hours or until necessary votes
> are reached.
>
> [ ] +1 Approve
>
> [ ] +0 No opinion
>
> [ ] -1 Disapprove (and reason why)
>
>
> Thanks,
> Sid
>
>
> List of all issues addressed in this release:
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
>
>
> Git tag for the release: https://github.com/apache/atlas/tree/release-2.2.0-rc0
>
> Sources for the release:
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz
>
> Source release verification:
>
> PGP Signature:
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.asc
>
> SHA512 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.sha512
>
> MD5 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.md5
>
>
> Keys to verify the signature of the release artifacts are available
> at: https://dist.apache.org/repos/dist/release/atlas/KEYS
>
>
> New features/enhancements:
>
> - Introduced Deferred Action - Tag propagation will be handled as
> a background process (AtlasTask)
>
> - Added Re-indexing as JAVA_PATCH
>
> - Created JavaPatch to add new super types to existing entities
>
> - Supported for Business Metadata in Atlas Export API
>
> - Supported HA for admin/task API
>
> - Enhanced export service to export Terms
>
> - Added new models and addition attributes to Atlas for Apache
> Flink, GCP and ADLS Gen 2
>
> - Added support to Configure HiveServer2 Hook to send Lineage-only Messages
>
> - Improved and enhanced Bulk Glossary Import supporting import
> with relations
>
> - Dependency between Atlas and Kafka Zookeeper removed
>
> - Improved performance of GraphHelper's guid and status getter methods
>
> - Added new privilege - "admin-audits" for Atlas Admin Audits authorization
>
> - Basic search Improvements - search text with special characters,
> optimize pagination
>
> - Updated free-text search processor to support Elasticsearch
>
> - Refactored and enhanced Atlas Python Client supporting Python 2.7
>
> - DSL Search enhancements - Support glossary terms and
> relationship, can handle null attribute values, Java Tinkerpop
> Traversals instead of GremlinScriptEngine, support search by
> classification and its attribute, Caching mechanism for translated dsl
> queries
>
> - Enhanced Export/Import Service to conditionally Support
> Simultaneous Operations
>
> - Enhanced NotificationHookConsumer for Large Message Processing
>
> - Enhanced Atlas authorization for Add/Update/Remove
> classification on entities
>
> - Provided option to add mandatory attribute to existing entity definition
>
> - Fixed Atlas UI to load faster in case of large number of
> classifications & entities
>
> - Updated versions of dependent libraries/components: JanusGraph,
> elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
> gremlin, Solr, groovy, netty, Kafka
>
>
Re: [VOTE] Release Apache Atlas version 2.2.0 - rc0
Posted by Madhan Neethiraj <ma...@apache.org>.
Sid,
Validated following on Atlas 2.2.0-rc0 - looks good overall! Found couple of issues given below; please review.
Thanks,
Madhan
1. successfully built Atlas using docker, for profile external-hbase-solr (instructions in dev-support/atlas-docker/README.txt)
2. successfully brought up Atlas in docker containers
3. successfully ran quick_start
4. search: performed basic search, advanced search, quick search
5. classification: performed add, associate, propagate, update, disassociate, delete
6. business-metadata: performed import, add/update/remove on entities, search
7. glossary: performed import, add/remove on entities, searched
8. verified basic operations on HBase hook (create table, drop table)
9. verified basic operations on Hive hook (create database/table/view, lineage, alter table, drop table/database)
10. verified entity audit entries for above operations
Issues found:
1. dev-support/atlas-docker/.env has BRANCH=branch-2.0; this should be updated to release-2.2.0-rc0, so that docker build would use Atlas sources from this tag
2. Atlas startup logs NoSuchFileException for ${atlas.log.dir}/atlas-metrics.out, which looks like incorrect metrics configuration in hadoop-metrics2.properties.
2021-08-05 07:54:07,353 WARN - [main:] ~ Error creating sink 'atlas-debug-metrics-context' (MetricsSystemImpl:508)
org.apache.hadoop.metrics2.impl.MetricsConfigException: Error creating plugin: org.apache.hadoop.metrics2.sink.FileSink
at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:210)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:531)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:503)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:479)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:62)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:58)
at org.apache.atlas.web.service.DebugMetricsWrapper.init(DebugMetricsWrapper.java:43)
...
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:110)
at org.apache.atlas.Atlas.main(Atlas.java:133)
Caused by: org.apache.hadoop.metrics2.MetricsException: Error creating ${atlas.log.dir}/atlas-metrics.out
at org.apache.hadoop.metrics2.sink.FileSink.init(FileSink.java:53)
at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:207)
... 63 more
Caused by: java.nio.file.NoSuchFileException: ${atlas.log.dir}/atlas-metrics.out
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at org.apache.hadoop.metrics2.sink.FileSink.init(FileSink.java:50)
... 64 more
On 8/4/21, 1:34 PM, "Sidharth Mishra" <si...@apache.org> wrote:
Atlas team,
Apache Atlas 2.2.0 release candidate #0 is now available for vote
within the dev community. Links to the release artifacts are given
below. Please review and vote.
The vote will be open for at least 72 hours or until necessary votes
are reached.
[ ] +1 Approve
[ ] +0 No opinion
[ ] -1 Disapprove (and reason why)
Thanks,
Sid
List of all issues addressed in this release:
https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20status%3DResolved%20AND%20fixVersion%3D2.2.0%20ORDER%20BY%20key%20DESC
Git tag for the release: https://github.com/apache/atlas/tree/release-2.2.0-rc0
Sources for the release:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz
Source release verification:
PGP Signature:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.asc
SHA512 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.sha512
MD5 Hash:
https://dist.apache.org/repos/dist/dev/atlas/2.2.0-rc0/apache-atlas-2.2.0-sources.tar.gz.md5
Keys to verify the signature of the release artifacts are available
at: https://dist.apache.org/repos/dist/release/atlas/KEYS
New features/enhancements:
- Introduced Deferred Action - Tag propagation will be handled as
a background process (AtlasTask)
- Added Re-indexing as JAVA_PATCH
- Created JavaPatch to add new super types to existing entities
- Supported for Business Metadata in Atlas Export API
- Supported HA for admin/task API
- Enhanced export service to export Terms
- Added new models and addition attributes to Atlas for Apache
Flink, GCP and ADLS Gen 2
- Added support to Configure HiveServer2 Hook to send Lineage-only Messages
- Improved and enhanced Bulk Glossary Import supporting import
with relations
- Dependency between Atlas and Kafka Zookeeper removed
- Improved performance of GraphHelper's guid and status getter methods
- Added new privilege - "admin-audits" for Atlas Admin Audits authorization
- Basic search Improvements - search text with special characters,
optimize pagination
- Updated free-text search processor to support Elasticsearch
- Refactored and enhanced Atlas Python Client supporting Python 2.7
- DSL Search enhancements - Support glossary terms and
relationship, can handle null attribute values, Java Tinkerpop
Traversals instead of GremlinScriptEngine, support search by
classification and its attribute, Caching mechanism for translated dsl
queries
- Enhanced Export/Import Service to conditionally Support
Simultaneous Operations
- Enhanced NotificationHookConsumer for Large Message Processing
- Enhanced Atlas authorization for Add/Update/Remove
classification on entities
- Provided option to add mandatory attribute to existing entity definition
- Fixed Atlas UI to load faster in case of large number of
classifications & entities
- Updated versions of dependent libraries/components: JanusGraph,
elasticsearch, JQuery, Http core, Http Client, slf4j, log4j, ant,
gremlin, Solr, groovy, netty, Kafka