You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by Wanyun Ding <di...@gmail.com> on 2022/12/15 06:59:19 UTC
Unable to Use "measure" configuration
Hi Griffin team,
I try to use Griffin spark-sql measurement to check data accuracy by follow the guide here which is a function optimized by you. But when I use “measure” keywords and try to save it, it will give me an error saying some bean file constraints issue. I did not find a place to submit issue so I write mail to you.
Look forward to your suggestions. Thanks in advanced.
https://github.com/apache/griffin/blob/master/griffin-doc/measure/measure-configuration-guide/sparksql.md <https://github.com/apache/griffin/blob/master/griffin-doc/measure/measure-configuration-guide/sparksql.md>
It would be nice if you could help provide a workable json sample. Did not find a useful one from https://github.com/apache/griffin/tree/master/measure/src/test/resources <https://github.com/apache/griffin/tree/master/measure/src/test/resources>
The griffin version we are using is griffin-0.6.0-source-release.zip. I only could submit the configuration
{
...
"measures": [
{
"name": "spark_sql_measure",
"type": "sparkSQL",
"data.source": "crime_report_source",
"config": {
"expr": "SELECT t.*, sq.zip IS NULL AS __measure_spark_sql_measure FROM crime_report_source AS t LEFT OUTER JOIN (SELECT zipcode as zip, COUNT(DISTINCT city) AS city_count FROM crime_report_source GROUP BY zipcode having city_count = 1) as sq ON sq.zip=t.zipcode",
"bad.record.definition": "__measure_spark_sql_measure"
},
"out": [
{
"type": "metric",
"name": "spark_sql_metric",
"flatten": "map"
},
{
"type": "record",
"name": "spark_sql_records"
}
]
}
]
...
}
<https://github.com/apache/griffin/blob/master/griffin-doc/measure/measure-configuration-guide/sparksql.md#key-parameters>
Re: Unable to Use "measure" configuration
Posted by William Guo <gu...@apache.org>.
hi Ding,
Thanks for reaching out.
Could you check the quickstart doc below for the workable solution?
https://griffin.apache.org/docs/quickstart.html
it looks like there are some validation errors before persisting, could you
show us the error log in the backend?
Thanks,
William
On Fri, Dec 16, 2022 at 3:58 PM Wanyun Ding <di...@gmail.com>
wrote:
> Hi Griffin team,
>
> I try to use Griffin spark-sql measurement to check data accuracy by
> follow the guide here which is a function optimized by you. But when I use
> “measure” keywords and try to save it, it will give me an error saying some
> bean file constraints issue. I did not find a place to submit issue so I
> write mail to you.
>
> Look forward to your suggestions. Thanks in advanced.
>
>
> https://github.com/apache/griffin/blob/master/griffin-doc/measure/measure-configuration-guide/sparksql.md
>
> It would be nice if you could help provide a workable json sample. Did not
> find a useful one from
> https://github.com/apache/griffin/tree/master/measure/src/test/resources
>
>
> The griffin version we are using is griffin-0.6.0-source-release.zip. I
> only could submit the configuration
>
> {
> ...
>
> "measures": [
> {
> "name": "spark_sql_measure",
> "type": "sparkSQL",
> "data.source": "crime_report_source",
> "config": {
> "expr": "SELECT t.*, sq.zip IS NULL AS __measure_spark_sql_measure FROM crime_report_source AS t LEFT OUTER JOIN (SELECT zipcode as zip, COUNT(DISTINCT city) AS city_count FROM crime_report_source GROUP BY zipcode having city_count = 1) as sq ON sq.zip=t.zipcode",
> "bad.record.definition": "__measure_spark_sql_measure"
> },
> "out": [
> {
> "type": "metric",
> "name": "spark_sql_metric",
> "flatten": "map"
> },
> {
> "type": "record",
> "name": "spark_sql_records"
> }
> ]
> }
> ]
>
> ...
> }
>
>
> <https://github.com/apache/griffin/blob/master/griffin-doc/measure/measure-configuration-guide/sparksql.md#key-parameters>
>
>
Re: Unable to Use "measure" configuration
Posted by William Guo <gu...@apache.org>.
hi Ding,
Thanks for reaching out.
Could you check the quickstart doc below for the workable solution?
https://griffin.apache.org/docs/quickstart.html
it looks like there are some validation errors before persisting, could you
show us the error log in the backend?
Thanks,
William
On Fri, Dec 16, 2022 at 3:58 PM Wanyun Ding <di...@gmail.com>
wrote:
> Hi Griffin team,
>
> I try to use Griffin spark-sql measurement to check data accuracy by
> follow the guide here which is a function optimized by you. But when I use
> “measure” keywords and try to save it, it will give me an error saying some
> bean file constraints issue. I did not find a place to submit issue so I
> write mail to you.
>
> Look forward to your suggestions. Thanks in advanced.
>
>
> https://github.com/apache/griffin/blob/master/griffin-doc/measure/measure-configuration-guide/sparksql.md
>
> It would be nice if you could help provide a workable json sample. Did not
> find a useful one from
> https://github.com/apache/griffin/tree/master/measure/src/test/resources
>
>
> The griffin version we are using is griffin-0.6.0-source-release.zip. I
> only could submit the configuration
>
> {
> ...
>
> "measures": [
> {
> "name": "spark_sql_measure",
> "type": "sparkSQL",
> "data.source": "crime_report_source",
> "config": {
> "expr": "SELECT t.*, sq.zip IS NULL AS __measure_spark_sql_measure FROM crime_report_source AS t LEFT OUTER JOIN (SELECT zipcode as zip, COUNT(DISTINCT city) AS city_count FROM crime_report_source GROUP BY zipcode having city_count = 1) as sq ON sq.zip=t.zipcode",
> "bad.record.definition": "__measure_spark_sql_measure"
> },
> "out": [
> {
> "type": "metric",
> "name": "spark_sql_metric",
> "flatten": "map"
> },
> {
> "type": "record",
> "name": "spark_sql_records"
> }
> ]
> }
> ]
>
> ...
> }
>
>
> <https://github.com/apache/griffin/blob/master/griffin-doc/measure/measure-configuration-guide/sparksql.md#key-parameters>
>
>