[jira] [Commented] (ATLAS-2889) S3 object tag import hook

["Ayush Nigam (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ATLAS-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626801#comment-16626801 ] 

Ayush Nigam commented on ATLAS-2889:
------------------------------------

Also I would like to highlight that S3 objects currently support AVRO types only according to S3 object data model for Atlas.

We will like to add types such as JSON,XML,CSV to current S3 object type definitions

[~toopt4] [~barbara]

> S3 object tag import hook
> -------------------------
>
>                 Key: ATLAS-2889
>                 URL: https://issues.apache.org/jira/browse/ATLAS-2889
>             Project: Atlas
>          Issue Type: New Feature
>          Components:  atlas-core, atlas-intg
>    Affects Versions: 1.1.0
>            Reporter: t oo
>            Priority: Major
>
> https://issues.apache.org/jira/browse/ATLAS-2708 introduced support for s3 type.
>  
> From the comments in the jira, there are some external steps needed to import object tags. This Jira is about creating the E2E functionality within Atlas. ie functionality in UI or an API that when supplied with argument of an s3 bucket/slice then Atlas automatically imports all the object names/tags recursively from under that s3 bucket/slice.
>  
> "It doesn't do it automatically through a listener like the hive hook.  We do it via lambda functions, triggered, say, on the creation of S3 object or pseudodirectory or bucket.  We package up the info into AtlasEntities and then publish to the ATLAS_HOOK kafka topic.
> You have to create your own Lambda code that creates AtlasEntities on the fly as trigerred by Lambda Function(on changes made to s3 object) and then push to Kafka Queue. This particular functionality is not part of Atlas tool as of now."
> cc: [~barbara] [~ayushmnnit]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)