epoll disconnections?

[Jacques Nadeau <ja...@dremio.com>]

Hey All,

If I recall correctly, many months ago Sudheesh discovered that we were
having instability in RPC connections in some situations due to bugs in the
epoll implementation that are fixed in a later version of Netty (~4.0.31?).
At the time, we shelved switching Netty because it also changed the memory
caching behavior (same thread to all thread) which seemed like a high risk
change. I thought that as part of this we decided the safest change was to
disable epoll RPC in our distribution. However, reviewing drill-env, it
doesn't look like we do this. See here [1].

Thoughts?

[1]
https://github.com/apache/drill/blob/master/distribution/src/resources/drill-env.sh#L19
--
Jacques Nadeau
CTO and Co-Founder, Dremio

Re: epoll disconnections?

[Hanifi Gunes <hg...@maprtech.com>]

Cool. Thanks.

On Fri, Mar 25, 2016 at 12:02 PM, Sudheesh Katkam <sk...@maprtech.com>
wrote:

> @Jacques, let me check the performance test [1] results from the past with
> epoll disabled (or we have to re-run).
>
> @Hanifi, With higher concurrency in performance testing, queries failed
> with this error:
>
> java.io.IOException: syscall:read(...)() failed: Connection reset by peer
>
> This is a known issue [2] with the version of Netty that Drill uses. I
> don’t think there is a specific ticket, but a search [3] shows a few
> relevant tickets.
>
> Thank you,
> Sudheesh
>
> [1] https://github.com/mapr/drill-perf-test-framework <
> https://github.com/mapr/drill-perf-test-framework>
> [2] https://github.com/netty/netty/issues/3539 <
> https://github.com/netty/netty/issues/3539>[3]
> https://issues.apache.org/jira/browse/DRILL-3119?jql=project%20%3D%20DRILL%20AND%20text%20~%20%22syscall%3Aread%22
> <
> https://issues.apache.org/jira/browse/DRILL-3119?jql=project%20=%20DRILL%20AND%20text%20~%20%22syscall:read%22
> >
>
> > On Mar 25, 2016, at 11:44 AM, Hanifi Gunes <hg...@maprtech.com> wrote:
> >
> > I am wondering what the issue and its manifestation was back then. Do we
> > have any JIRAs created for this before?
> >
> >
> > Thanks.
> > -Hanifi
> >
> > On Fri, Mar 25, 2016 at 9:57 AM, Jacques Nadeau <ja...@dremio.com>
> wrote:
> >
> >> Hey All,
> >>
> >> If I recall correctly, many months ago Sudheesh discovered that we were
> >> having instability in RPC connections in some situations due to bugs in
> the
> >> epoll implementation that are fixed in a later version of Netty
> (~4.0.31?).
> >> At the time, we shelved switching Netty because it also changed the
> memory
> >> caching behavior (same thread to all thread) which seemed like a high
> risk
> >> change. I thought that as part of this we decided the safest change was
> to
> >> disable epoll RPC in our distribution. However, reviewing drill-env, it
> >> doesn't look like we do this. See here [1].
> >>
> >> Thoughts?
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/drill/blob/master/distribution/src/resources/drill-env.sh#L19
> >> --
> >> Jacques Nadeau
> >> CTO and Co-Founder, Dremio
> >>
>
>

Re: epoll disconnections?

[Sudheesh Katkam <sk...@maprtech.com>]

@Jacques, let me check the performance test [1] results from the past with epoll disabled (or we have to re-run).

@Hanifi, With higher concurrency in performance testing, queries failed with this error:

java.io.IOException: syscall:read(...)() failed: Connection reset by peer

This is a known issue [2] with the version of Netty that Drill uses. I don’t think there is a specific ticket, but a search [3] shows a few relevant tickets.

Thank you,
Sudheesh

[1] https://github.com/mapr/drill-perf-test-framework <https://github.com/mapr/drill-perf-test-framework>
[2] https://github.com/netty/netty/issues/3539 <https://github.com/netty/netty/issues/3539>[3] https://issues.apache.org/jira/browse/DRILL-3119?jql=project%20%3D%20DRILL%20AND%20text%20~%20%22syscall%3Aread%22 <https://issues.apache.org/jira/browse/DRILL-3119?jql=project%20=%20DRILL%20AND%20text%20~%20%22syscall:read%22>

> On Mar 25, 2016, at 11:44 AM, Hanifi Gunes <hg...@maprtech.com> wrote:
> 
> I am wondering what the issue and its manifestation was back then. Do we
> have any JIRAs created for this before?
> 
> 
> Thanks.
> -Hanifi
> 
> On Fri, Mar 25, 2016 at 9:57 AM, Jacques Nadeau <ja...@dremio.com> wrote:
> 
>> Hey All,
>> 
>> If I recall correctly, many months ago Sudheesh discovered that we were
>> having instability in RPC connections in some situations due to bugs in the
>> epoll implementation that are fixed in a later version of Netty (~4.0.31?).
>> At the time, we shelved switching Netty because it also changed the memory
>> caching behavior (same thread to all thread) which seemed like a high risk
>> change. I thought that as part of this we decided the safest change was to
>> disable epoll RPC in our distribution. However, reviewing drill-env, it
>> doesn't look like we do this. See here [1].
>> 
>> Thoughts?
>> 
>> [1]
>> 
>> https://github.com/apache/drill/blob/master/distribution/src/resources/drill-env.sh#L19
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>> 

Re: epoll disconnections?

[Hanifi Gunes <hg...@maprtech.com>]

I am wondering what the issue and its manifestation was back then. Do we
have any JIRAs created for this before?


Thanks.
-Hanifi

On Fri, Mar 25, 2016 at 9:57 AM, Jacques Nadeau <ja...@dremio.com> wrote:

> Hey All,
>
> If I recall correctly, many months ago Sudheesh discovered that we were
> having instability in RPC connections in some situations due to bugs in the
> epoll implementation that are fixed in a later version of Netty (~4.0.31?).
> At the time, we shelved switching Netty because it also changed the memory
> caching behavior (same thread to all thread) which seemed like a high risk
> change. I thought that as part of this we decided the safest change was to
> disable epoll RPC in our distribution. However, reviewing drill-env, it
> doesn't look like we do this. See here [1].
>
> Thoughts?
>
> [1]
>
> https://github.com/apache/drill/blob/master/distribution/src/resources/drill-env.sh#L19
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>