You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2017/04/18 15:29:41 UTC
[jira] [Assigned] (SPARK-20368) Support Sentry on PySpark workers
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-20368:
------------------------------------
Assignee: (was: Apache Spark)
> Support Sentry on PySpark workers
> ---------------------------------
>
> Key: SPARK-20368
> URL: https://issues.apache.org/jira/browse/SPARK-20368
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
> Affects Versions: 2.1.0
> Reporter: Alexander Shorin
>
> [Setry|https://sentry.io] is a well known among Python developers system to capture, classify, track and explain tracebacks, helping people better understand what went wrong, how to reproduce the issue and fix it.
> Any Spark application on Python is actually divided on two parts:
> 1. The one that runs on "driver side". That part user may control in all the ways it want and provide reports to Sentry is very easy to do here.
> 2. The one that runs on executors. That's Python UDFs and the rest transformation functions. Unfortunately, here we cannot provide such kind of feature. And that is the part this feature is about.
> In order to simplify developing experience, it would be nice to have optional Sentry support on PySpark worker level.
> What this feature could looks like?
> 1. PySpark will have new extra named {{sentry}} which installs Sentry client and the rest required things if are necessary. This is an optional install-time dependency.
> 2. PySpark worker will be able to detect presence of Sentry support and send error reports there.
> 3. All configuration of Sentry could and will be done via standard Sentry`s environment variables.
> What this feature will give to users?
> 1. Better exceptions in Sentry. From driver-side application, now all of them get recorded as like `Py4JJavaError` where the real executor exception is written in a traceback body.
> 2. Greater simplification of understanding context when thing went wrong and why.
> 3. Simplify Python UDFs debug and issues reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org