[jira] [Comment Edited] (GRIFFIN-316) Spark runtime exception cannot be caught while running a dq application

["Chitral Verma (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/GRIFFIN-316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17005218#comment-17005218 ] 

Chitral Verma edited comment on GRIFFIN-316 at 12/30/19 9:23 AM:
-----------------------------------------------------------------

ok great!

 

thanks


was (Author: chitralverma):
ok great!

> Spark runtime exception cannot be caught while running a dq application
> -----------------------------------------------------------------------
>
>                 Key: GRIFFIN-316
>                 URL: https://issues.apache.org/jira/browse/GRIFFIN-316
>             Project: Griffin
>          Issue Type: Bug
>    Affects Versions: 0.4.0, 0.5.0, 0.6.0
>            Reporter: Yu LIU
>            Priority: Major
>             Fix For: 0.6.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> If we put an invalid rule for a batch job (as it happens quite often given that the rules are evaluated at runtime via spark sql), the exception thrown by SparkSession has not been caught and transferred properly to user via "Try" instance, but the job actually succeed with a "Success" returned.
> The reason is that we are only wrapping the returned Boolean result by applying "Try" at the most outside level for DQApp.run, so the exception thrown deeper through the call stack cannot be caught.
>  
> Here is an example config file to reproduce the issue:
> {noformat}
> {
>   "name": "prof_batch",
>   "process.type": "batch",
>   "timestamp": 123456,
>   "data.sources": [
>     {
>       "name": "source",
>       "connectors": [
>         {
>           "type": "avro",
>           "version": "1.7",
>           "dataframe.name" : "this_table",
>           "config": {
>             "file.name": "src/test/resources/users_info_src.avro"
>           },
>           "pre.proc": [
>             {
>               "dsl.type": "spark-sql",
>               "rule": "select * from this_table where user_id < 10014"
>             }
>           ]
>         }
>       ]
>     }
>   ],
>   "evaluate.rule": {
>     "rules": [
>       {
>         "dsl.type": "griffin-dsl",
>         "dq.type": "profiling",
>         "out.dataframe.name": "prof",
>         "rule": "xxx",
>         "out":[
>           {
>             "type": "metric",
>             "name": "prof",
>             "flatten": "array"
>           }
>         ]
>       }
>     ]
>   },
>   "sinks": ["CONSOLE"]
> }{noformat}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)