Error reporting/collecting for users

[Tobias Pfeiffer <tg...@preferred.jp>]

Hi,

in my Spark Streaming application, computations depend on users' input in
terms of
 * user-defined functions
 * computation rules
 * etc.
that can throw exceptions in various cases (think: exception in UDF,
division by zero, invalid access by key etc.).

Now I am wondering about what is a good/reasonable way to deal with those
errors. I think I want to continue processing (the whole stream processing
pipeline should not die because of one single malformed item in the
stream), i.e., catch the exception, but still I need a way to tell the user
something went wrong.

So how can I get the information that something went wrong back to the
driver and what is a reasonable way to do that?

While writing this, something like the following came into my mind:
  val errors = sc.accumulator(...) // of type List[Throwable]
  dstream.map(item => {
    Try {
      someUdf(item)
    } match {
      case Success(value) =>
        value
      case Failure(err) =>
        errors += err  // remember error
        0  // default value
    }
  })
Does this make sense?

Thanks
Tobias

Re: Error reporting/collecting for users

[Tathagata Das <ta...@gmail.com>]

You could use foreachRDD to do the operations and then inside the
foreach create an accumulator to gather all the errors together

dstream.foreachRDD { rdd =>

   val accumulator = new Accumulator[....]

   rdd.map { ..... }.count   // whatever operation that is error prone

   // gather all errors from  accumulator and show on webpage, push to
database?
}


On Tue, Jan 27, 2015 at 9:34 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:
> Hi,
>
> On Wed, Jan 28, 2015 at 1:45 PM, Soumitra Kumar <ku...@gmail.com>
> wrote:
>>
>> It is a Streaming application, so how/when do you plan to access the
>> accumulator on driver?
>
>
> Well... maybe there would be some user command or web interface showing the
> errors that have happened during processing...?
>
> Thanks
> Tobias
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Error reporting/collecting for users

[Tobias Pfeiffer <tg...@preferred.jp>]

Hi,

On Wed, Jan 28, 2015 at 1:45 PM, Soumitra Kumar <ku...@gmail.com>
wrote:

> It is a Streaming application, so how/when do you plan to access the
> accumulator on driver?
>

Well... maybe there would be some user command or web interface showing the
errors that have happened during processing...?

Thanks
Tobias

Re: Error reporting/collecting for users

[Soumitra Kumar <ku...@gmail.com>]

It is a Streaming application, so how/when do you plan to access the
accumulator on driver?

On Tue, Jan 27, 2015 at 6:48 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:

> Hi,
>
> thanks for your mail!
>
> On Wed, Jan 28, 2015 at 11:44 AM, Tathagata Das <
> tathagata.das1565@gmail.com> wrote:
>
>> That seems reasonable to me. Are you having any problems doing it this
>> way?
>>
>
> Well, actually I haven't done that yet. The idea of using accumulators to
> collect errors just came while writing the email, but I thought I'd just
> keep writing and see if anyone has any other suggestions ;-)
>
> Thanks
> Tobias
>
>

Re: Error reporting/collecting for users

[Tobias Pfeiffer <tg...@preferred.jp>]

Hi,

thanks for your mail!

On Wed, Jan 28, 2015 at 11:44 AM, Tathagata Das <tathagata.das1565@gmail.com
> wrote:

> That seems reasonable to me. Are you having any problems doing it this way?
>

Well, actually I haven't done that yet. The idea of using accumulators to
collect errors just came while writing the email, but I thought I'd just
keep writing and see if anyone has any other suggestions ;-)

Thanks
Tobias