You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Kai Chen (Jira)" <ji...@apache.org> on 2019/08/23 07:09:00 UTC
[jira] [Commented] (ATLAS-3130) Atlas stops reporting CTAS tables
once an invalid relationshipDef is created
[ https://issues.apache.org/jira/browse/ATLAS-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914002#comment-16914002 ]
Kai Chen commented on ATLAS-3130:
---------------------------------
{code:json}
{
"version":{
"version":"1.0.0",
"versionParts":[
1
]
},
"msgCompressionKind":"NONE",
"msgSplitIdx":1,
"msgSplitCount":1,
"msgCreationTime":1566459683140,
"message":{
"type":"ENTITY_CREATE_V2",
"user":"root",
"entities":{
"referredEntities":{
"-1748077198110670":{
"typeName":"sqoop_dbdatastore",
"attributes":{
"owner":"root",
"storeUri":"jdbc:mysql://127.0.0.1:3306/razzil?zeroDateTimeBehavior=convertToNull",
"dbStoreType":"mysql",
"qualifiedName":"mysql --url jdbc:mysql://127.0.0.1:3306/razzil?zeroDateTimeBehavior=convertToNull --database test",
"name":"mysql --url jdbc:mysql://127.0.0.1:3306/razzil?zeroDateTimeBehavior=convertToNull --database test",
"description":"",
"source":"sys_persistent_token",
"storeUse":"TABLE"
},
"guid":"-1748077198110670",
"provenanceType":0,
"version":0,
"proxy":false
},
"-1748077198110671":{
"typeName":"hive_db",
"attributes":{
"qualifiedName":"test@primary",
"clusterName":"primary",
"name":"test"
},
"guid":"-1748077198110671",
"provenanceType":0,
"version":0,
"proxy":false
}
},
"entities":[
{
"typeName":"sqoop_process",
"attributes":{
"commandlineOpts":{
"reset.onemapper":"false",
"codegen.output.delimiters.enclose":"0",
"sqlconnection.metadata.transaction.isolation.level":"2",
"codegen.input.delimiters.escape":"0",
"codegen.auto.compile.dir":"true",
"accumulo.batch.size":"10240000",
"codegen.input.delimiters.field":"0",
"accumulo.create.table":"false",
"mainframe.input.dataset.type":"p",
"enable.compression":"false",
"skip.dist.cache":"false",
"accumulo.max.latency":"5000",
"db.username":"root",
"sqoop.throwOnError":"false",
"db.clear.staging.table":"false",
"codegen.input.delimiters.enclose":"0",
"hive.database.name":"test",
"hdfs.append.dir":"false",
"import.direct.split.size":"0",
"hcatalog.drop.and.create.table":"false",
"import.all_tables.exclude":"zd_reg_certificate_file,zd_reg_certificate_info",
"codegen.output.delimiters.record":"10",
"codegen.output.delimiters.field":"1",
"hbase.bulk.load.enabled":"false",
"mapreduce.num.mappers":"1",
"export.new.update":"UpdateOnly",
"db.require.password":"true",
"hive.import":"true",
"customtool.options.jsonmap":"{}",
"hdfs.delete-target.dir":"false",
"codegen.output.delimiters.enclose.required":"false",
"direct.import":"false",
"codegen.output.dir":".",
"hdfs.file.format":"TextFile",
"hive.drop.delims":"false",
"codegen.input.delimiters.record":"0",
"db.batch":"false",
"split.limit":"null",
"hcatalog.create.table":"false",
"hive.fail.table.exists":"false",
"hive.overwrite.table":"true",
"incremental.mode":"None",
"temporary.dirRoot":"_sqoop",
"verbose":"false",
"import.max.inline.lob.size":"16777216",
"import.fetch.size":"-2147483648",
"codegen.input.delimiters.enclose.required":"false",
"relaxed.isolation":"false",
"sqoop.oracle.escaping.disabled":"true",
"hbase.create.table":"false",
"codegen.compile.dir":"/tmp/sqoop-root/compile/60d07986ffda505aae69e7c90d7f057f",
"codegen.output.delimiters.escape":"0",
"db.connect.string":"jdbc:mysql://127.0.0.1:3306/razzil?zeroDateTimeBehavior=convertToNull"
},
"qualifiedName":"sqoop import --connect jdbc:mysql://127.0.0.1:3306/razzil?zeroDateTimeBehavior=convertToNull --database test--hive-import --hive-database test --hive-cluster primary",
"name":"sqoop import --connect jdbc:mysql://127.0.0.1:3306/razzil?zeroDateTimeBehavior=convertToNull --database test--hive-import --hive-database test --hive-cluster primary",
"startTime":1566292892036,
"endTime":1566292907628,
"userName":"root",
"operation":"import"
},
"guid":"-1748077198110672",
"provenanceType":0,
"version":0,
"relationshipAttributes":{
"outputs":[
{
"guid":"-1748077198110671",
"typeName":"hive_db"
}
],
"inputs":[
{
"guid":"-1748077198110670",
"typeName":"sqoop_dbdatastore"
}
]
},
"proxy":false
}
]
}
}
}
{code}
> Atlas stops reporting CTAS tables once an invalid relationshipDef is created
> -----------------------------------------------------------------------------
>
> Key: ATLAS-3130
> URL: https://issues.apache.org/jira/browse/ATLAS-3130
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Reporter: Umesh Padashetty
> Assignee: Merryle Wang
> Priority: Minor
>
> Created an invalid relationshipDef with the following body
> {code:java}
> {
> "relationshipDefs": [{
> "propagateTags": "ONE_TO_TWO",
> "description": "default relationshipDef description with name: sample_dcef_KcFcR3xIN",
> "relationshipCategory": "ASSOCIATION",
> "typeVersion": "1.0",
> "attributeDefs": [],
> "endDef2": {
> "name": "inputs",
> "isLegacyAttribute": false,
> "isContainer": false,
> "cardinality": "SINGLE",
> "type": "Process",
> "description": "default relationshipEndDef description with name: inputs"
> },
> "endDef1": {
> "name": "hive_table_to_process",
> "isLegacyAttribute": false,
> "isContainer": false,
> "cardinality": "SET",
> "type": "hive_table",
> "description": "default relationshipEndDef description with name: hive_table_to_process"
> },
> "guid": "-302256315524233986",
> "name": "sample_dcef_KcFcR3xIN"
> }]
> }{code}
> After this, all the CTAS queries start failing with the following error
> {code:java}
> 2019-04-09 17:47:38,992 ERROR - [NotificationHookConsumer thread-0:] ~ graph rollback due to exception (GraphTransactionInterceptor:154)
> org.apache.atlas.exception.AtlasBaseException: invalid relationshipDef: sample_dcef_KcFcR3xIN: end type 1: Process, end type 2: hive_column_lineage{code}
> The tables are created in the hive, but they are not reported in Atlas. Creation of such an invalid relationshipDef should ideally be blocked.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)