jira ticket BEAM-10990

[Steven Gaunt <st...@equifax.com>]

Hi guys

I've created a jira ticket BEAM-10990
<https://issues.apache.org/jira/browse/BEAM-10990> as we are keen to use
ElasticSearchIO.

would there be any objections if I was create a branch and propose a fix
for BEAM-10990 <https://issues.apache.org/jira/browse/BEAM-10990> ?

thanks

Steve

-- 
This message contains proprietary information from Equifax which may be 
confidential. If you are not an intended recipient, please refrain from any 
disclosure, copying, distribution or use of this information and note that 
such actions are prohibited. If you have received this transmission in 
error, please notify by e-mail postmaster@equifax.com 
<ma...@equifax.com>. Equifax® is a registered trademark of 
Equifax Inc. All rights reserved.

Re: jira ticket BEAM-10990

[Pablo Estrada <pa...@google.com>]

Hi Steven,
I've added you as a contributor, so feel free to assign the issue to
yourself.

BigQueryIO has a similar pattern for its write transform. You can do
BigQueryIO.write(.......).withExtendedErrorInfo()[1], and it will return a
WriteResult[2], which contains a PCollection of failed inserts with their
errors[3].

You could add a similar option to ElasticSearchIO which determines when to
retry errors, and what to do with them. Ideally, the default behavior would
not change (to ensure backwards compatibility), but when specifying the new
configuration, that would help you address your use case.
By the way, this pattern is very common, so I bet it will be a useful
addition for other users.

If that works for you, we'd be happy to welcome your contribution.
Best
-P.

[1]
https://beam.apache.org/releases/javadoc/2.24.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withExtendedErrorInfo--
[2]
https://beam.apache.org/releases/javadoc/2.24.0/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.html

[3]
https://beam.apache.org/releases/javadoc/2.24.0/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.html#getFailedInsertsWithErr--

On Wed, Sep 30, 2020 at 11:23 AM Steven Gaunt <st...@equifax.com>
wrote:

> Hi guys
>
> I've created a jira ticket BEAM-10990
> <https://issues.apache.org/jira/browse/BEAM-10990> as we are keen to use
> ElasticSearchIO.
>
> would there be any objections if I was create a branch and propose a fix
> for BEAM-10990 <https://issues.apache.org/jira/browse/BEAM-10990> ?
>
> thanks
>
> Steve
>
> This message contains proprietary information from Equifax which may be
> confidential. If you are not an intended recipient, please refrain from any
> disclosure, copying, distribution or use of this information and note that
> such actions are prohibited. If you have received this transmission in
> error, please notify by e-mail postmaster@equifax.com. Equifax® is a
> registered trademark of Equifax Inc. All rights reserved.
>
>