You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by Preetam Shingavi <ps...@expediagroup.com.INVALID> on 2020/02/24 22:23:26 UTC
Spark sql output sink to Elastic search - No fields in result value
Hi,
I have a hive table and want to try running a spark-sql and output metric result to elastic search. Right now I am facing an issue to see the projected columns from the spark-sql query.
Here is the DQ job:
{
"name":"fms_count_score_measure",
"measure.type":"griffin",
"process.type":"BATCH",
"owner":"test",
"description":"fms count measure description",
"data.sources":[
{
"name":"fms_ip",
"connectors":[
{
"name":"fms_ip_connector",
"type":"HIVE",
"version":"1.2",
"data.unit":"1hour",
"data.time.zone":"UTC(WET,GMT)",
"config":{
"database":"default",
"table.name":"fms",
"where":"me_type='ITEM_PUBLISHED'"
}
}
]
}
],
"evaluate.rule":{
"rules":[
{
"dsl.type":"spark-sql",
"name":"fms_count_score",
"rule":"select count(*) as `ip_count` from fms_ip",
"out.dataframe.name": "fms_ip_cnt",
"out": [
{
"type": "metric",
"name": "fms_ip_cnt"
}
]
}
]
},
"sinks": [
"CONSOLE",
"ELASTICSEARCH",
"HDFS"
]
}
The metrics written is: {"name":"FMS_Comp_score_Job","tmst":1582578720000,"value":{}}
I expected ‘ip_count’ to be a part of the value field but cannot see why it’s not working as expected.
Any suggestions, highly appreciated 😊
Thank you in advance!
Regards,
Preetam
Re: Spark sql output sink to Elastic search - No fields in result
value
Posted by Preetam Shingavi <ps...@expediagroup.com.INVALID>.
+ Users
From: Preetam Shingavi <ps...@expediagroup.com>
Date: Monday, February 24, 2020 at 2:23 PM
To: "dev@griffin.apache.org" <de...@griffin.apache.org>
Subject: Spark sql output sink to Elastic search - No fields in result value
Hi,
I have a hive table and want to try running a spark-sql and output metric result to elastic search. Right now I am facing an issue to see the projected columns from the spark-sql query.
Here is the DQ job:
{
"name":"fms_count_score_measure",
"measure.type":"griffin",
"process.type":"BATCH",
"owner":"test",
"description":"fms count measure description",
"data.sources":[
{
"name":"fms_ip",
"connectors":[
{
"name":"fms_ip_connector",
"type":"HIVE",
"version":"1.2",
"data.unit":"1hour",
"data.time.zone":"UTC(WET,GMT)",
"config":{
"database":"default",
"table.name":"fms",
"where":"me_type='ITEM_PUBLISHED'"
}
}
]
}
],
"evaluate.rule":{
"rules":[
{
"dsl.type":"spark-sql",
"name":"fms_count_score",
"rule":"select count(*) as `ip_count` from fms_ip",
"out.dataframe.name": "fms_ip_cnt",
"out": [
{
"type": "metric",
"name": "fms_ip_cnt"
}
]
}
]
},
"sinks": [
"CONSOLE",
"ELASTICSEARCH",
"HDFS"
]
}
The metrics written is: {"name":"FMS_Comp_score_Job","tmst":1582578720000,"value":{}}
I expected ‘ip_count’ to be a part of the value field but cannot see why it’s not working as expected.
Any suggestions, highly appreciated 😊
Thank you in advance!
Regards,
Preetam
Re: Spark sql output sink to Elastic search - No fields in result
value
Posted by Preetam Shingavi <ps...@expediagroup.com.INVALID>.
+ Users
From: Preetam Shingavi <ps...@expediagroup.com>
Date: Monday, February 24, 2020 at 2:23 PM
To: "dev@griffin.apache.org" <de...@griffin.apache.org>
Subject: Spark sql output sink to Elastic search - No fields in result value
Hi,
I have a hive table and want to try running a spark-sql and output metric result to elastic search. Right now I am facing an issue to see the projected columns from the spark-sql query.
Here is the DQ job:
{
"name":"fms_count_score_measure",
"measure.type":"griffin",
"process.type":"BATCH",
"owner":"test",
"description":"fms count measure description",
"data.sources":[
{
"name":"fms_ip",
"connectors":[
{
"name":"fms_ip_connector",
"type":"HIVE",
"version":"1.2",
"data.unit":"1hour",
"data.time.zone":"UTC(WET,GMT)",
"config":{
"database":"default",
"table.name":"fms",
"where":"me_type='ITEM_PUBLISHED'"
}
}
]
}
],
"evaluate.rule":{
"rules":[
{
"dsl.type":"spark-sql",
"name":"fms_count_score",
"rule":"select count(*) as `ip_count` from fms_ip",
"out.dataframe.name": "fms_ip_cnt",
"out": [
{
"type": "metric",
"name": "fms_ip_cnt"
}
]
}
]
},
"sinks": [
"CONSOLE",
"ELASTICSEARCH",
"HDFS"
]
}
The metrics written is: {"name":"FMS_Comp_score_Job","tmst":1582578720000,"value":{}}
I expected ‘ip_count’ to be a part of the value field but cannot see why it’s not working as expected.
Any suggestions, highly appreciated 😊
Thank you in advance!
Regards,
Preetam