You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Oran Shuster <or...@houzz.com> on 2021/10/12 22:06:12 UTC
I/O reactor status: STOPPED after moving to elasticsearch7 connector
Flink version 1.13.1
ES Version 7.12.0
Flink deployment type - session cluster
Scala version 2.12
Same job used to run with elasticsearch6 connector with about the same load and had no issues
Since moving to elasticsearch7 more and more exceptions were being thrown about reactor being stopped in our logs
[IllegalStateException] with message [Request cannot be executed; I/O reactor status: STOPPED] with rest status -1
Looking online for this type of error gave me some results from people instantiating the ES client on their own which is irrelevant for our case. Also found that it might be caused by an uncaught exception in the failure handler, so i wrapped all the code in try..catch but found no uncaught exceptions
Next thing I tried was to force using the updated version of ES rest client with the same version as my cluster 7.12. That didn't seem to fix any issues
Problem is that once this happens it does not fix itself. That instance of the sink will continuously get those exceptions until we restart the job. I tried to look for any logs before that but couldn't find anything
Re: I/O reactor status: STOPPED after moving to elasticsearch7 connector
Posted by Arvid Heise <ar...@apache.org>.
Hi Oran,
could you check if smaller batches improve the situation?
On Sat, Oct 16, 2021 at 2:15 AM Oran Shuster <or...@houzz.com> wrote:
> The cluster is not really overloaded and also couldn't find some ES errors
> logs (atleast something that is repeating)
> The job IS very busy (100% on the sink and backpressure) but I think its
> due to the reactor exceptions being thrown
>
> what do you mean by incorrect batching?
>
> On 2021/10/13 07:43:38, Itamar Syn-Hershko <it...@bigdataboutique.com>
> wrote:
> > Hi Oran, can you check your ES logs / metrics?
> >
> > Most issues we see with the ES sink are around incorrect batching and/or
> > overloaded clusters. Could it be your ES write queue is building up?
> >
> > On Wed, Oct 13, 2021 at 1:06 AM Oran Shuster <or...@houzz.com>
> wrote:
> >
> > > Flink version 1.13.1
> > > ES Version 7.12.0
> > > Flink deployment type - session cluster
> > > Scala version 2.12
> > >
> > > Same job used to run with elasticsearch6 connector with about the same
> > > load and had no issues
> > > Since moving to elasticsearch7 more and more exceptions were being
> thrown
> > > about reactor being stopped in our logs
> > >
> > > [IllegalStateException] with message [Request cannot be executed; I/O
> > > reactor status: STOPPED] with rest status -1
> > >
> > > Looking online for this type of error gave me some results from people
> > > instantiating the ES client on their own which is irrelevant for our
> case.
> > > Also found that it might be caused by an uncaught exception in the
> failure
> > > handler, so i wrapped all the code in try..catch but found no uncaught
> > > exceptions
> > >
> > > Next thing I tried was to force using the updated version of ES rest
> > > client with the same version as my cluster 7.12. That didn't seem to
> fix
> > > any issues
> > >
> > > Problem is that once this happens it does not fix itself. That
> instance of
> > > the sink will continuously get those exceptions until we restart the
> job. I
> > > tried to look for any logs before that but couldn't find anything
> > >
> >
> >
> > --
> > [image: logo] <https://bigdataboutique.com/>
> > Itamar Syn-Hershko
> > CTO, Founder
> > Email: itamar@bigdataboutique.com
> > Website: https://bigdataboutique.com
> > <https://twitter.com/synhershko> <https://github.com/synhershko>
> > <https://www.linkedin.com/in/itamar-syn-hershko-78b25013/>
> > <https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>
> >
>
Re: I/O reactor status: STOPPED after moving to elasticsearch7 connector
Posted by Oran Shuster <or...@houzz.com>.
The cluster is not really overloaded and also couldn't find some ES errors logs (atleast something that is repeating)
The job IS very busy (100% on the sink and backpressure) but I think its due to the reactor exceptions being thrown
what do you mean by incorrect batching?
On 2021/10/13 07:43:38, Itamar Syn-Hershko <it...@bigdataboutique.com> wrote:
> Hi Oran, can you check your ES logs / metrics?
>
> Most issues we see with the ES sink are around incorrect batching and/or
> overloaded clusters. Could it be your ES write queue is building up?
>
> On Wed, Oct 13, 2021 at 1:06 AM Oran Shuster <or...@houzz.com> wrote:
>
> > Flink version 1.13.1
> > ES Version 7.12.0
> > Flink deployment type - session cluster
> > Scala version 2.12
> >
> > Same job used to run with elasticsearch6 connector with about the same
> > load and had no issues
> > Since moving to elasticsearch7 more and more exceptions were being thrown
> > about reactor being stopped in our logs
> >
> > [IllegalStateException] with message [Request cannot be executed; I/O
> > reactor status: STOPPED] with rest status -1
> >
> > Looking online for this type of error gave me some results from people
> > instantiating the ES client on their own which is irrelevant for our case.
> > Also found that it might be caused by an uncaught exception in the failure
> > handler, so i wrapped all the code in try..catch but found no uncaught
> > exceptions
> >
> > Next thing I tried was to force using the updated version of ES rest
> > client with the same version as my cluster 7.12. That didn't seem to fix
> > any issues
> >
> > Problem is that once this happens it does not fix itself. That instance of
> > the sink will continuously get those exceptions until we restart the job. I
> > tried to look for any logs before that but couldn't find anything
> >
>
>
> --
> [image: logo] <https://bigdataboutique.com/>
> Itamar Syn-Hershko
> CTO, Founder
> Email: itamar@bigdataboutique.com
> Website: https://bigdataboutique.com
> <https://twitter.com/synhershko> <https://github.com/synhershko>
> <https://www.linkedin.com/in/itamar-syn-hershko-78b25013/>
> <https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>
>
Re: I/O reactor status: STOPPED after moving to elasticsearch7 connector
Posted by Itamar Syn-Hershko <it...@bigdataboutique.com>.
Hi Oran, can you check your ES logs / metrics?
Most issues we see with the ES sink are around incorrect batching and/or
overloaded clusters. Could it be your ES write queue is building up?
On Wed, Oct 13, 2021 at 1:06 AM Oran Shuster <or...@houzz.com> wrote:
> Flink version 1.13.1
> ES Version 7.12.0
> Flink deployment type - session cluster
> Scala version 2.12
>
> Same job used to run with elasticsearch6 connector with about the same
> load and had no issues
> Since moving to elasticsearch7 more and more exceptions were being thrown
> about reactor being stopped in our logs
>
> [IllegalStateException] with message [Request cannot be executed; I/O
> reactor status: STOPPED] with rest status -1
>
> Looking online for this type of error gave me some results from people
> instantiating the ES client on their own which is irrelevant for our case.
> Also found that it might be caused by an uncaught exception in the failure
> handler, so i wrapped all the code in try..catch but found no uncaught
> exceptions
>
> Next thing I tried was to force using the updated version of ES rest
> client with the same version as my cluster 7.12. That didn't seem to fix
> any issues
>
> Problem is that once this happens it does not fix itself. That instance of
> the sink will continuously get those exceptions until we restart the job. I
> tried to look for any logs before that but couldn't find anything
>
--
[image: logo] <https://bigdataboutique.com/>
Itamar Syn-Hershko
CTO, Founder
Email: itamar@bigdataboutique.com
Website: https://bigdataboutique.com
<https://twitter.com/synhershko> <https://github.com/synhershko>
<https://www.linkedin.com/in/itamar-syn-hershko-78b25013/>
<https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>