[QUESTION] retrial in sources by the runners

[Etienne Chauchot <ec...@apache.org>]

Hi all,

I have a question concerning retrial of sources. I've looked at the code of direct runner and spark runner on bounded
sources. As far as I can tell, if there is a failure in reading a record from the reader of the source, there will be no
retrial from the runner, there will just be an exception thrown. The only retrial that can take place with the source is
the one that could be done by the reader itself.

Can you confirm that I did not miss something?

Etienne

Re: [QUESTION] retrial in sources by the runners

[Lukasz Cwik <lc...@google.com>]

Runners are responsible for retry semantics, they should catch the failure
and choose whether they want to retry or not. I think your reading of the
code is correct.

Some I/O layers do retry but that is more about attempting to continue
processing within a bundle instead of failing and having the runner choose
to retry the bundle or not.

On Thu, Aug 30, 2018 at 5:57 AM Etienne Chauchot <ec...@apache.org>
wrote:

> Hi all,
>
> I have a question concerning retrial of sources. I've looked at the code
> of direct runner and spark runner on bounded sources. As far as I can tell,
> if there is a failure in reading a record from the reader of the source,
> there will be no retrial from the runner, there will just be an exception
> thrown. The only retrial that can take place with the source is the one
> that could be done by the reader itself.
>
> Can you confirm that I did not miss something?
>
> Etienne
>