You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Paresh Devalia <pa...@freestoneinfotech.com> on 2022/02/23 10:17:35 UTC
Review Request 73862: ATLAS-3682 Multivalued attributes are not listed under Business Metadata Attribute filter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73862/
-----------------------------------------------------------
Review request for atlas, Jayendra Parab and Pinal Shah.
Bugs: ATLAS-3682
https://issues.apache.org/jira/browse/ATLAS-3682
Repository: atlas
Description
-------
Problem : Multivalued attributes like arrayList, HashMap are not searchable.
On clicking on atlas UI basic search, attribute like arrayList and hashmap are not listed in attribute filter list because on atlas server end there is no proper mapping of atlas typeName "ArrayList" and "HashMap" with respect to janusgraph. Janusgraph does not support arrayList or HashMap typeName.
Current Behaviour : Atlas server has hardcoding of Cardinality as SINGLE for attribute typeName = ArrayList.
Approach : To support typeName arrayList<String>, janusgraph expects typeName as String, cardinality as LIST/SET. hence we need to map internal Atlas Types accordingly
Changes Needed:
- Changes in creation of attribute of type arrayList
- Changes in setting the values to the property
- Changes in getting the values of the property
- Java Patch to be added to update the cardinality of the attribute whose typeName is ArrayList to SINGLE in the typeDefinition because internally for ArrayList it is hardcoded to SINGLE
- TypeName hashmap attribute will get stored in "||" double pipe format.
- Changes in filter search for atlas typeName "ArrayList" and "HashMap" from UI or POST request.
Diffs
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 2669c8a7e
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java ef8bd4265
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusElement.java 66ee4e96e
graphdb/janus/src/main/java/org/janusgraph/diskstorage/solr/Solr6Index.java 81ae202f1
intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 21ce23657
repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java f69dc4215
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java d65bb1a2a
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java d5112bbf6
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java 2fce123f8
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 6d8305a32
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 794847555
repository/src/main/java/org/apache/atlas/util/SearchPredicateUtil.java 9c2975645
Diff: https://reviews.apache.org/r/73862/diff/1/
Testing
-------
Tested for above mentioned scenario, working as expected.
Step to follow for testing this scenario,
-mvn clean install
-make cardinality = single changes in 1030-hive_model.json file wherein entity=hive_table and attributeDefs name=aliases
-atlas_start
-quick_start
1) Create an attributeDefs for hive_table entity
PUT http method -> URL api/atlas/v2/types/typedefs
{ "name": "dummylist",
"typeName": "array<string>",
"isOptional": true,
"cardinality": "SET",
"valuesMinCount": 0,
"valuesMaxCount": 2147483647,
"isUnique": false,
"isIndexable": true,
"includeInNotification": false,
"searchWeight": -1
},
{
"name": "dummyMap",
"typeName": "map<string,string>",
"isOptional": true,
"cardinality": "SET",
"valuesMinCount": 0,
"valuesMaxCount": 1,
"isUnique": false,
"isIndexable": true,
"includeInNotification": false,
"searchWeight": -1
},
2) create an hive_table entity from UI or POST request
POST http method -> URL api/atlas/v2/entity
{
"entity": {
"guid": "-1",
"isProxy": false,
"isIncomplete": false,
"provenanceType": 0,
"version": 0,
"relationshipAttributes": {
"columns": [],
"ddlQueries": [],
"inputToProcesses": [],
"meanings": [],
"outputFromProcesses": [],
"partitionKeys": [],
"schema": []
},
"typeName": "hive_table",
"attributes": {
"aliases": [
"X",
"Y",
"Z",
"A"
],
"createTime": 1638210600000,
"lastAccessTime": 1638210600000,
"name": "A1",
"dummyMap": {
"name": "dummy"
},
"dummyList": [
"X",
"Y",
"Z",
"A"
],
"qualifiedName": "A1",
"replicatedFrom": [],
"replicatedTo": []
}
},
"referredEntities": {}
}
3) Search list through POST API
POST http method -> api/atlas/v2/search/basic
SEARCH OPERATOR SUPPORT :- =, IN, isNull, notNull, notEmpty
LIST SEARCH API :-
{
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"entityFilters": {
"condition": "AND",
"criterion": [
{
"attributeName": "dummyList",
"operator": "=",
"attributeValue": "x"
}
]
},
"tagFilters": null,
"attributes": [
"dummyList"
],
"limit": 25,
"offset": 0,
"typeName": "hive_table",
"classification": null,
"termName": null
}
------------------------------------------------------------------------------------
MAP SEARCH API :-
{
"excludeDeletedEntities": true,
"includeSubClassifications": true,
"includeSubTypes": true,
"includeClassificationAttributes": true,
"entityFilters": {
"condition": "AND",
"criterion": [
{
"attributeName": "dummyMap
"operator": "=",
"attributeValue": "name=dummy"
}
]
},
"tagFilters": null,
"attributes": [
"dummyMap"
],
"limit": 25,
"offset": 0,
"typeName": "hive_table",
"classification": null,
"termName": null
}
Thanks,
Paresh Devalia