You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Josh Schlesser <jo...@spoutable.com> on 2016/02/02 08:51:19 UTC
setting session params in a rest request
First: Im a total newb at contributing to apache projects so please excuse any indiscretions, feel free to give comments on style or whatever, i take feedback well. Thick skin too.
Ill give some background next and then a proposal.
Background:
I recently changed over to using authentication in the 1.5 snapshot because I need to have a session via the REST api so that I can set the session storage options in an initial query for a subsequent CTAS query. Previously all rest calls seemed to be completely independent.
Since the change I have started seeing ‘too many files open’ errors in my drillbit.log and the drillbit java process becomes effectively hung waiting for open file descriptor slots. When running the top command the machine is running at max load due to the drillbit process and the drillbit becomes effectively unresponsive, even the simple pages in the web console don’t respond. Investigating further it seems that there might be a file kept open per session by the drillbit process for the life of the session. I used the lsof unix command on the drillbit process and found a lot of unix pipes. Looking at the code it looks like these pipes could be for the communication between the web process and the rpc server, with one being allocated per session. I haven’t validated this, its just a guess after scanning the code. I had 1.4 running without this requirement and without ever seeing the error. It seems without authentication the number of open files is a non-issue for me, possibly due to sessions.
I'm wondering if my guess about what is causing the ‘too many open files’ error is plausible? Does anybody with a deeper understanding of the architecture have any comments on this?
Proposal:
Assuming sessions are the issue, I am making some changes to my rest client so that sessions are more effectively used and I can up the ulimit for the drillbit process for the linux user in hopes of mitigating this. I am effectively creating a rest client based session pool that resets session variables to defaults when the session gets reused. However, it seems hacky.
Below is an idea for getting per request based settings which seems less hacky in the long term.
Can I add a new array member to the query.json REST method in a backwards compatible way to set session level parameters in a single request? Currently a rest request via the api has a body like so:
{ “queryType”: “SQL”, “query” : “<drill query>”}
id like to do the following
{ “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”: [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
or even
{ “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”: [“SET `option_name` = value”, “SET `option_name1` = value1”,“SET `option_name2` = value2”, “SET `option_name3` = value3”]}
As far as I can tell drill is essentially stateless between queries right now except for session level system parameters and authentication. There aren’t any in memory temp tables or cursors or variables like PL/SQL or PSQL or other SQLs that would make it stateful.
Given the stateless assumption, being able to set session level params on a per request basis would cover all of the cases that I might need. It looks relatively straight forward to add something to QueryWrapper to accept an optional query session settings section of the json packet and execute those ’SET' commands before the final query. This will work for me, as I can run without authentication in an ’secure' backend environment which will remove sessions and hence file descriptors, assuming my assumptions about file descriptors and sessions are correct.
My java is rusty (circa 2003) but some casual googling implies that if this were added as a 3rd @FormParam to submitQuery in QueryResources it would be magically be null if it werent present and could easily be ignored. If its present then an alternative constructor of QueryWrapper could be called with the extra param and it would be easy to alter its run method to execute the SET commands. There would need to be some error handling of course if the SET commands were illegal or failed to run for some reason.
If this seems reasonable, how do I go about contributing? I looked through the links in the docs to apache foundation incubator projects but the links to drill were broken :( http://drill.apache.org/team.html <http://drill.apache.org/team.html> I read this http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <http://drill.apache.org/docs/apache-drill-contribution-guidelines/> and i have subscribed to the dev mailing list (obvious since you are getting this). It said to post here before creating a JIRA. Am I missing anything in my assumptions? Comments? Should I just submit a JIRA and a patch or submit a JIRA and a comment or wait for comments before coding stuff up as an example?
Thanks for taking the time to read and respond.
Josh
Re: setting session params in a rest request
Posted by Venki Korukanti <ve...@gmail.com>.
Currently there is no limit. It is basically depends on jetty's
HashSessionManager. A quick glance through the HashSessionManager doesn't
reveal any API to control the max number of sessions. I will keep looking.
I just fixed the issue of resource leak. Resources are not released
properly when they are expired. I will log a bug in few mins. I think we
should get that into the 1.5 release.
Thanks
Venki
On Thu, Feb 4, 2016 at 1:27 PM, Jason Altekruse <al...@gmail.com>
wrote:
> Venki,
>
> Even for authenticated sessions, do we impose any kind of upper limit on
> the number of concurrent connections? I seems like we should issue a
> reasonable error message about an over-used Drillbit before the process
> dies due to hitting the system file handle limit.
>
> - Jason
>
> On Thu, Feb 4, 2016 at 11:00 AM, Josh Schlesser <jo...@spoutable.com>
> wrote:
>
> > Thanks Venki,
> > Let me know how I can help further.
> >
> > > On Feb 2, 2016, at 10:23 PM, Venki Korukanti <
> venki.korukanti@gmail.com>
> > wrote:
> > >
> > > I think you made valid points. It makes sense to have session less REST
> > > calls in both auth enabled and disabled cases.
> > >
> > > In case of auth enabled:
> > > 1) Session-less calls can be authenticated using Basic auth (this was
> > > already asked on mailing list sometime back) as a start and move onto
> > token
> > > based auth later. These requests usually come from non-browsers. The
> only
> > > issue is setting options before the query. For this we can implement
> your
> > > suggestion of enhancing the query REST API to accept the options.
> > > 2) Session-based call using form auth for browser based access. If we
> > > enhance the UI to enter options in the query form, we don't need any
> > > session on server actually.
> > >
> > > I will get a fix ASAP to remove the sessions in anonymous calls as they
> > the
> > > session are not reused in non-browser cases.
> > >
> > > Thanks
> > > Venki
> > >
> > > On Tue, Feb 2, 2016 at 9:20 PM, Josh Schlesser <jo...@spoutable.com>
> > wrote:
> > >
> > >> No, it wasn’t logging out, it was just stopping, obviously that caused
> > >> dangling sessions for the authenticated scenario.
> > >>
> > >> I don’t think that a short timeout for anonymous sessions is a good
> way
> > to
> > >> go for anonymous api calls. Session management isn’t what anybody
> would
> > >> expect when using a REST api that is anonymous in a server to server
> > >> context. I would expect to use a token for authorization for a
> server
> > to
> > >> server REST api as well. I’m not saying that is what it should be
> here,
> > >> but that is what my general expectation is based on using other apis.
> > In
> > >> the case of browser to server REST apis, I have run into
> authentication
> > for
> > >> a browser session and subsequent REST calls leaning on a browser
> cookie
> > for
> > >> persistent authentication.
> > >>
> > >> Removing sessions for anonymous calls seems like the right path and
> > >> possibly easy and I think would be the expected behavior from most
> > >> developers. I would advocate for sessionless and token authenticated
> > REST
> > >> apis for when using authentication for the server to server case and
> > cookie
> > >> based with a session for the browser to server scenario, but its
> really
> > the
> > >> browser that has a session, not the api per se, its just piggybacking
> > on a
> > >> regular authenticated web session for the REST api calls.
> > >>
> > >> This would actually leave me in a quandary for what I am trying to do
> > >> which is set a session configuration option ’store.format', but I cant
> > >> think of any reason that those types of settings shouldn’t just be set
> > on a
> > >> per request basis for a REST api. In a server to server context for a
> > rest
> > >> api, keeping it sessionless means you could front a cluster of
> drillbits
> > >> with a load balancer and not worry about dying nodes and sticky
> sessions
> > >> etc...
> > >>
> > >> I have to get something up and running quickly right now so im
> > versioning
> > >> back to 1.4 and just spinning up a separate drillbit that will have
> the
> > >> store.format system variable set to ‘json’ . it will be ok for me
> until
> > a
> > >> good long term solution arrives in drill.
> > >>
> > >> I’ll run the test on short session_max_idle_secs to 30 seconds on
> > >> 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation
> > >> problem, but keep in mind that means that users of the web console
> will
> > >> have 30 seconds between pages or they have to authenticate again,
> which
> > >> will probably be very annoying. It doesnt seem like a good long term
> > >> solution either.
> > >>
> > >> How do you think all of this should work? I look forward to staying
> > >> involved.
> > >>
> > >> Cheers,
> > >> Josh
> > >>
> > >>> On Feb 2, 2016, at 4:40 PM, Venki Korukanti <
> venki.korukanti@gmail.com
> > >
> > >> wrote:
> > >>>
> > >>> When auth is *enabled*, is the worker process logging out after
> queries
> > >> are
> > >>> done? When auth is *disabled* can you set session_max_idle_secs in
> > >>> drill.exec.http block in drill-override.conf to something like 30
> > (secs)
> > >>> and try? This way anonymous sessions are closed quickly and not kept
> > for
> > >>> 1hr (default value). I think we may need to avoid creating sessions
> in
> > >>> anonymous mode (when auth is disabled).
> > >>>
> > >>> Thanks
> > >>> Venki
> > >>>
> > >>> On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <jo...@spoutable.com>
> > >> wrote:
> > >>>
> > >>>> I have a background worker process (on a server, not a browser) that
> > >> kicks
> > >>>> off every minute or so and issues some queries sequentially to the
> > rest
> > >>>> query endpoint. In 1.4 with no authentication this worked fine
> > except
> > >>>> that in 1 instance I need to issue a CTAS query with a different
> > format
> > >>>> (json).
> > >>>>
> > >>>> I upgraded to 1.5-SNAPSHOT commit
> > >> bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
> > >>>>
> > >>>> Since the upgrade I am getting a resource starvation problem with or
> > >>>> without authentication
> > >>>> The drillbit process stays up for a an hour or less and then becomes
> > >>>> unresponsive and eats up the cpu.
> > >>>>
> > >>>> It is definitely a resource starvation issue, not sure if its a
> > resource
> > >>>> leak.
> > >>>> Below is a stack trace.
> > >>>> Also when i lsof on the pid there are a lot (more than a thousand)
> of
> > >>>> files like this listed which are used by NIO selectors. so it
> smells
> > >> like
> > >>>> a resource leak.
> > >>>>
> > >>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
> > NAME
> > >>>> java 2931 root 288u 0000 0,11 0 7705
> > >>>> anon_inode
> > >>>>
> > >>>> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
> > >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> > >>>> java.lang.IllegalStateException: failed to create a child event loop
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> > >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> > >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> > >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> > >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> > >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> > >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> > >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too
> > many
> > >>>> open files
> > >>>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> > >>>> ... 25 common frames omitted
> > >>>> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
> > >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> > >>>> java.lang.IllegalStateException: failed to create a child event loop
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> > >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> > >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> > >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> > >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> > >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> > >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> > >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> > >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at
> > >>>>
> > >>
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> > >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> > >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> > >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too
> > many
> > >>>> open files
> > >>>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> > >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> > >>>> at
> > >>>>
> > >>
> >
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> > >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> > >>>> ... 25 common frames omitted
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <
> > venki.korukanti@gmail.com
> > >>>
> > >>>> wrote:
> > >>>>>
> > >>>>> Currently we keep the DrillClient per session. All the state is in
> > >> Server
> > >>>>> and DrillClient is the reference to reuse the state. DrillClient is
> > >>>>> automatically closed when the session expires (default value is 1hr
> > >> after
> > >>>>> the last activity on session) or user explicitly logs out. I am
> > trying
> > >> to
> > >>>>> understand if there is a resource leak. Do you have too many
> sessions
> > >>>> open
> > >>>>> when the system load is max or just few sessions but you have
> already
> > >> ran
> > >>>>> many queries using the existing sessions? If it is the former it is
> > >>>>> understandable to have per connection per session life. Also are
> the
> > >>>>> resources not freeing up after logout?
> > >>>>>
> > >>>>> If you need to have multiple simultaneous sessions, it is better to
> > >>>> connect
> > >>>>> to different Drillbits (may be in a round-robin fashion) than
> always
> > >>>>> connecting to a single Drillbit.
> > >>>>>
> > >>>>> Thanks
> > >>>>> Venki
> > >>>>>
> > >>>>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <
> josh@spoutable.com
> > >>>> <ma...@spoutable.com>> wrote:
> > >>>>>
> > >>>>>> First: Im a total newb at contributing to apache projects so
> please
> > >>>> excuse
> > >>>>>> any indiscretions, feel free to give comments on style or
> whatever,
> > i
> > >>>> take
> > >>>>>> feedback well. Thick skin too.
> > >>>>>>
> > >>>>>>
> > >>>>>> Ill give some background next and then a proposal.
> > >>>>>>
> > >>>>>> Background:
> > >>>>>> I recently changed over to using authentication in the 1.5
> snapshot
> > >>>>>> because I need to have a session via the REST api so that I can
> set
> > >> the
> > >>>>>> session storage options in an initial query for a subsequent CTAS
> > >> query.
> > >>>>>> Previously all rest calls seemed to be completely independent.
> > >>>>>>
> > >>>>>> Since the change I have started seeing ‘too many files open’
> errors
> > in
> > >>>> my
> > >>>>>> drillbit.log and the drillbit java process becomes effectively
> hung
> > >>>> waiting
> > >>>>>> for open file descriptor slots. When running the top command the
> > >>>> machine
> > >>>>>> is running at max load due to the drillbit process and the
> drillbit
> > >>>> becomes
> > >>>>>> effectively unresponsive, even the simple pages in the web console
> > >> don’t
> > >>>>>> respond. Investigating further it seems that there might be a
> file
> > >>>> kept
> > >>>>>> open per session by the drillbit process for the life of the
> > session.
> > >>>> I
> > >>>>>> used the lsof unix command on the drillbit process and found a lot
> > of
> > >>>> unix
> > >>>>>> pipes. Looking at the code it looks like these pipes could be for
> > the
> > >>>>>> communication between the web process and the rpc server, with one
> > >> being
> > >>>>>> allocated per session. I haven’t validated this, its just a guess
> > >> after
> > >>>>>> scanning the code. I had 1.4 running without this requirement
> and
> > >>>> without
> > >>>>>> ever seeing the error. It seems without authentication the number
> > of
> > >>>> open
> > >>>>>> files is a non-issue for me, possibly due to sessions.
> > >>>>>>
> > >>>>>> I'm wondering if my guess about what is causing the ‘too many open
> > >>>> files’
> > >>>>>> error is plausible? Does anybody with a deeper understanding of
> > the
> > >>>>>> architecture have any comments on this?
> > >>>>>>
> > >>>>>> Proposal:
> > >>>>>> Assuming sessions are the issue, I am making some changes to my
> rest
> > >>>>>> client so that sessions are more effectively used and I can up the
> > >>>> ulimit
> > >>>>>> for the drillbit process for the linux user in hopes of mitigating
> > >>>> this. I
> > >>>>>> am effectively creating a rest client based session pool that
> resets
> > >>>>>> session variables to defaults when the session gets reused.
> > >> However,
> > >>>> it
> > >>>>>> seems hacky.
> > >>>>>>
> > >>>>>> Below is an idea for getting per request based settings which
> seems
> > >> less
> > >>>>>> hacky in the long term.
> > >>>>>>
> > >>>>>> Can I add a new array member to the query.json REST method in a
> > >>>> backwards
> > >>>>>> compatible way to set session level parameters in a single
> request?
> > >>>>>> Currently a rest request via the api has a body like so:
> > >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”}
> > >>>>>>
> > >>>>>> id like to do the following
> > >>>>>>
> > >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”,
> “sessionSettings”:
> > >>>>>> [“option_1_name”:”option_1_value”,
> > “option_2_name”:”option_2_value”]}
> > >>>>>>
> > >>>>>> or even
> > >>>>>>
> > >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”,
> “sessionSettings”:
> > >>>> [“SET
> > >>>>>> `option_name` = value”, “SET `option_name1` = value1”,“SET
> > >>>> `option_name2` =
> > >>>>>> value2”, “SET `option_name3` = value3”]}
> > >>>>>>
> > >>>>>> As far as I can tell drill is essentially stateless between
> queries
> > >>>> right
> > >>>>>> now except for session level system parameters and authentication.
> > >>>> There
> > >>>>>> aren’t any in memory temp tables or cursors or variables like
> PL/SQL
> > >> or
> > >>>>>> PSQL or other SQLs that would make it stateful.
> > >>>>>>
> > >>>>>> Given the stateless assumption, being able to set session level
> > params
> > >>>> on
> > >>>>>> a per request basis would cover all of the cases that I might
> need.
> > >> It
> > >>>>>> looks relatively straight forward to add something to QueryWrapper
> > to
> > >>>>>> accept an optional query session settings section of the json
> packet
> > >> and
> > >>>>>> execute those ’SET' commands before the final query. This will
> > work
> > >>>> for
> > >>>>>> me, as I can run without authentication in an ’secure' backend
> > >>>> environment
> > >>>>>> which will remove sessions and hence file descriptors, assuming my
> > >>>>>> assumptions about file descriptors and sessions are correct.
> > >>>>>>
> > >>>>>>
> > >>>>>> My java is rusty (circa 2003) but some casual googling implies
> that
> > if
> > >>>>>> this were added as a 3rd @FormParam to submitQuery in
> QueryResources
> > >> it
> > >>>>>> would be magically be null if it werent present and could easily
> be
> > >>>>>> ignored. If its present then an alternative constructor of
> > >> QueryWrapper
> > >>>>>> could be called with the extra param and it would be easy to alter
> > its
> > >>>> run
> > >>>>>> method to execute the SET commands. There would need to be some
> > error
> > >>>>>> handling of course if the SET commands were illegal or failed to
> run
> > >> for
> > >>>>>> some reason.
> > >>>>>>
> > >>>>>> If this seems reasonable, how do I go about contributing? I
> looked
> > >>>>>> through the links in the docs to apache foundation incubator
> > projects
> > >>>> but
> > >>>>>> the links to drill were broken :(
> > http://drill.apache.org/team.html
> > >> <
> > >>>>>> http://drill.apache.org/team.html <
> > http://drill.apache.org/team.html
> > >>>>
> > >>>> I read this
> > >>>>>>
> http://drill.apache.org/docs/apache-drill-contribution-guidelines/
> > <
> > >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
> <
> > >>>>>>
> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
> > >>>> and
> > >>>>>> i have subscribed to the dev mailing list (obvious since you are
> > >> getting
> > >>>>>> this). It said to post here before creating a JIRA. Am I
> missing
> > >>>>>> anything in my assumptions? Comments? Should I just submit a
> JIRA
> > >> and
> > >>>> a
> > >>>>>> patch or submit a JIRA and a comment or wait for comments before
> > >> coding
> > >>>>>> stuff up as an example?
> > >>>>>>
> > >>>>>> Thanks for taking the time to read and respond.
> > >>>>>>
> > >>>>>> Josh
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Re: setting session params in a rest request
Posted by Jason Altekruse <al...@gmail.com>.
Venki,
Even for authenticated sessions, do we impose any kind of upper limit on
the number of concurrent connections? I seems like we should issue a
reasonable error message about an over-used Drillbit before the process
dies due to hitting the system file handle limit.
- Jason
On Thu, Feb 4, 2016 at 11:00 AM, Josh Schlesser <jo...@spoutable.com> wrote:
> Thanks Venki,
> Let me know how I can help further.
>
> > On Feb 2, 2016, at 10:23 PM, Venki Korukanti <ve...@gmail.com>
> wrote:
> >
> > I think you made valid points. It makes sense to have session less REST
> > calls in both auth enabled and disabled cases.
> >
> > In case of auth enabled:
> > 1) Session-less calls can be authenticated using Basic auth (this was
> > already asked on mailing list sometime back) as a start and move onto
> token
> > based auth later. These requests usually come from non-browsers. The only
> > issue is setting options before the query. For this we can implement your
> > suggestion of enhancing the query REST API to accept the options.
> > 2) Session-based call using form auth for browser based access. If we
> > enhance the UI to enter options in the query form, we don't need any
> > session on server actually.
> >
> > I will get a fix ASAP to remove the sessions in anonymous calls as they
> the
> > session are not reused in non-browser cases.
> >
> > Thanks
> > Venki
> >
> > On Tue, Feb 2, 2016 at 9:20 PM, Josh Schlesser <jo...@spoutable.com>
> wrote:
> >
> >> No, it wasn’t logging out, it was just stopping, obviously that caused
> >> dangling sessions for the authenticated scenario.
> >>
> >> I don’t think that a short timeout for anonymous sessions is a good way
> to
> >> go for anonymous api calls. Session management isn’t what anybody would
> >> expect when using a REST api that is anonymous in a server to server
> >> context. I would expect to use a token for authorization for a server
> to
> >> server REST api as well. I’m not saying that is what it should be here,
> >> but that is what my general expectation is based on using other apis.
> In
> >> the case of browser to server REST apis, I have run into authentication
> for
> >> a browser session and subsequent REST calls leaning on a browser cookie
> for
> >> persistent authentication.
> >>
> >> Removing sessions for anonymous calls seems like the right path and
> >> possibly easy and I think would be the expected behavior from most
> >> developers. I would advocate for sessionless and token authenticated
> REST
> >> apis for when using authentication for the server to server case and
> cookie
> >> based with a session for the browser to server scenario, but its really
> the
> >> browser that has a session, not the api per se, its just piggybacking
> on a
> >> regular authenticated web session for the REST api calls.
> >>
> >> This would actually leave me in a quandary for what I am trying to do
> >> which is set a session configuration option ’store.format', but I cant
> >> think of any reason that those types of settings shouldn’t just be set
> on a
> >> per request basis for a REST api. In a server to server context for a
> rest
> >> api, keeping it sessionless means you could front a cluster of drillbits
> >> with a load balancer and not worry about dying nodes and sticky sessions
> >> etc...
> >>
> >> I have to get something up and running quickly right now so im
> versioning
> >> back to 1.4 and just spinning up a separate drillbit that will have the
> >> store.format system variable set to ‘json’ . it will be ok for me until
> a
> >> good long term solution arrives in drill.
> >>
> >> I’ll run the test on short session_max_idle_secs to 30 seconds on
> >> 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation
> >> problem, but keep in mind that means that users of the web console will
> >> have 30 seconds between pages or they have to authenticate again, which
> >> will probably be very annoying. It doesnt seem like a good long term
> >> solution either.
> >>
> >> How do you think all of this should work? I look forward to staying
> >> involved.
> >>
> >> Cheers,
> >> Josh
> >>
> >>> On Feb 2, 2016, at 4:40 PM, Venki Korukanti <venki.korukanti@gmail.com
> >
> >> wrote:
> >>>
> >>> When auth is *enabled*, is the worker process logging out after queries
> >> are
> >>> done? When auth is *disabled* can you set session_max_idle_secs in
> >>> drill.exec.http block in drill-override.conf to something like 30
> (secs)
> >>> and try? This way anonymous sessions are closed quickly and not kept
> for
> >>> 1hr (default value). I think we may need to avoid creating sessions in
> >>> anonymous mode (when auth is disabled).
> >>>
> >>> Thanks
> >>> Venki
> >>>
> >>> On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <jo...@spoutable.com>
> >> wrote:
> >>>
> >>>> I have a background worker process (on a server, not a browser) that
> >> kicks
> >>>> off every minute or so and issues some queries sequentially to the
> rest
> >>>> query endpoint. In 1.4 with no authentication this worked fine
> except
> >>>> that in 1 instance I need to issue a CTAS query with a different
> format
> >>>> (json).
> >>>>
> >>>> I upgraded to 1.5-SNAPSHOT commit
> >> bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
> >>>>
> >>>> Since the upgrade I am getting a resource starvation problem with or
> >>>> without authentication
> >>>> The drillbit process stays up for a an hour or less and then becomes
> >>>> unresponsive and eats up the cpu.
> >>>>
> >>>> It is definitely a resource starvation issue, not sure if its a
> resource
> >>>> leak.
> >>>> Below is a stack trace.
> >>>> Also when i lsof on the pid there are a lot (more than a thousand) of
> >>>> files like this listed which are used by NIO selectors. so it smells
> >> like
> >>>> a resource leak.
> >>>>
> >>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE
> NAME
> >>>> java 2931 root 288u 0000 0,11 0 7705
> >>>> anon_inode
> >>>>
> >>>> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
> >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >>>> java.lang.IllegalStateException: failed to create a child event loop
> >>>> at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>> at
> >>>>
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >>>> at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too
> many
> >>>> open files
> >>>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>> ... 25 common frames omitted
> >>>> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
> >>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >>>> java.lang.IllegalStateException: failed to create a child event loop
> >>>> at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>> at
> >>>>
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >>>> at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at
> >>>>
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too
> many
> >>>> open files
> >>>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >>>> at
> >>>>
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >>>> ... 25 common frames omitted
> >>>>
> >>>>
> >>>>
> >>>>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <
> venki.korukanti@gmail.com
> >>>
> >>>> wrote:
> >>>>>
> >>>>> Currently we keep the DrillClient per session. All the state is in
> >> Server
> >>>>> and DrillClient is the reference to reuse the state. DrillClient is
> >>>>> automatically closed when the session expires (default value is 1hr
> >> after
> >>>>> the last activity on session) or user explicitly logs out. I am
> trying
> >> to
> >>>>> understand if there is a resource leak. Do you have too many sessions
> >>>> open
> >>>>> when the system load is max or just few sessions but you have already
> >> ran
> >>>>> many queries using the existing sessions? If it is the former it is
> >>>>> understandable to have per connection per session life. Also are the
> >>>>> resources not freeing up after logout?
> >>>>>
> >>>>> If you need to have multiple simultaneous sessions, it is better to
> >>>> connect
> >>>>> to different Drillbits (may be in a round-robin fashion) than always
> >>>>> connecting to a single Drillbit.
> >>>>>
> >>>>> Thanks
> >>>>> Venki
> >>>>>
> >>>>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com
> >>>> <ma...@spoutable.com>> wrote:
> >>>>>
> >>>>>> First: Im a total newb at contributing to apache projects so please
> >>>> excuse
> >>>>>> any indiscretions, feel free to give comments on style or whatever,
> i
> >>>> take
> >>>>>> feedback well. Thick skin too.
> >>>>>>
> >>>>>>
> >>>>>> Ill give some background next and then a proposal.
> >>>>>>
> >>>>>> Background:
> >>>>>> I recently changed over to using authentication in the 1.5 snapshot
> >>>>>> because I need to have a session via the REST api so that I can set
> >> the
> >>>>>> session storage options in an initial query for a subsequent CTAS
> >> query.
> >>>>>> Previously all rest calls seemed to be completely independent.
> >>>>>>
> >>>>>> Since the change I have started seeing ‘too many files open’ errors
> in
> >>>> my
> >>>>>> drillbit.log and the drillbit java process becomes effectively hung
> >>>> waiting
> >>>>>> for open file descriptor slots. When running the top command the
> >>>> machine
> >>>>>> is running at max load due to the drillbit process and the drillbit
> >>>> becomes
> >>>>>> effectively unresponsive, even the simple pages in the web console
> >> don’t
> >>>>>> respond. Investigating further it seems that there might be a file
> >>>> kept
> >>>>>> open per session by the drillbit process for the life of the
> session.
> >>>> I
> >>>>>> used the lsof unix command on the drillbit process and found a lot
> of
> >>>> unix
> >>>>>> pipes. Looking at the code it looks like these pipes could be for
> the
> >>>>>> communication between the web process and the rpc server, with one
> >> being
> >>>>>> allocated per session. I haven’t validated this, its just a guess
> >> after
> >>>>>> scanning the code. I had 1.4 running without this requirement and
> >>>> without
> >>>>>> ever seeing the error. It seems without authentication the number
> of
> >>>> open
> >>>>>> files is a non-issue for me, possibly due to sessions.
> >>>>>>
> >>>>>> I'm wondering if my guess about what is causing the ‘too many open
> >>>> files’
> >>>>>> error is plausible? Does anybody with a deeper understanding of
> the
> >>>>>> architecture have any comments on this?
> >>>>>>
> >>>>>> Proposal:
> >>>>>> Assuming sessions are the issue, I am making some changes to my rest
> >>>>>> client so that sessions are more effectively used and I can up the
> >>>> ulimit
> >>>>>> for the drillbit process for the linux user in hopes of mitigating
> >>>> this. I
> >>>>>> am effectively creating a rest client based session pool that resets
> >>>>>> session variables to defaults when the session gets reused.
> >> However,
> >>>> it
> >>>>>> seems hacky.
> >>>>>>
> >>>>>> Below is an idea for getting per request based settings which seems
> >> less
> >>>>>> hacky in the long term.
> >>>>>>
> >>>>>> Can I add a new array member to the query.json REST method in a
> >>>> backwards
> >>>>>> compatible way to set session level parameters in a single request?
> >>>>>> Currently a rest request via the api has a body like so:
> >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”}
> >>>>>>
> >>>>>> id like to do the following
> >>>>>>
> >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> >>>>>> [“option_1_name”:”option_1_value”,
> “option_2_name”:”option_2_value”]}
> >>>>>>
> >>>>>> or even
> >>>>>>
> >>>>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> >>>> [“SET
> >>>>>> `option_name` = value”, “SET `option_name1` = value1”,“SET
> >>>> `option_name2` =
> >>>>>> value2”, “SET `option_name3` = value3”]}
> >>>>>>
> >>>>>> As far as I can tell drill is essentially stateless between queries
> >>>> right
> >>>>>> now except for session level system parameters and authentication.
> >>>> There
> >>>>>> aren’t any in memory temp tables or cursors or variables like PL/SQL
> >> or
> >>>>>> PSQL or other SQLs that would make it stateful.
> >>>>>>
> >>>>>> Given the stateless assumption, being able to set session level
> params
> >>>> on
> >>>>>> a per request basis would cover all of the cases that I might need.
> >> It
> >>>>>> looks relatively straight forward to add something to QueryWrapper
> to
> >>>>>> accept an optional query session settings section of the json packet
> >> and
> >>>>>> execute those ’SET' commands before the final query. This will
> work
> >>>> for
> >>>>>> me, as I can run without authentication in an ’secure' backend
> >>>> environment
> >>>>>> which will remove sessions and hence file descriptors, assuming my
> >>>>>> assumptions about file descriptors and sessions are correct.
> >>>>>>
> >>>>>>
> >>>>>> My java is rusty (circa 2003) but some casual googling implies that
> if
> >>>>>> this were added as a 3rd @FormParam to submitQuery in QueryResources
> >> it
> >>>>>> would be magically be null if it werent present and could easily be
> >>>>>> ignored. If its present then an alternative constructor of
> >> QueryWrapper
> >>>>>> could be called with the extra param and it would be easy to alter
> its
> >>>> run
> >>>>>> method to execute the SET commands. There would need to be some
> error
> >>>>>> handling of course if the SET commands were illegal or failed to run
> >> for
> >>>>>> some reason.
> >>>>>>
> >>>>>> If this seems reasonable, how do I go about contributing? I looked
> >>>>>> through the links in the docs to apache foundation incubator
> projects
> >>>> but
> >>>>>> the links to drill were broken :(
> http://drill.apache.org/team.html
> >> <
> >>>>>> http://drill.apache.org/team.html <
> http://drill.apache.org/team.html
> >>>>
> >>>> I read this
> >>>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/
> <
> >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> <
> >>>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
> >>>> and
> >>>>>> i have subscribed to the dev mailing list (obvious since you are
> >> getting
> >>>>>> this). It said to post here before creating a JIRA. Am I missing
> >>>>>> anything in my assumptions? Comments? Should I just submit a JIRA
> >> and
> >>>> a
> >>>>>> patch or submit a JIRA and a comment or wait for comments before
> >> coding
> >>>>>> stuff up as an example?
> >>>>>>
> >>>>>> Thanks for taking the time to read and respond.
> >>>>>>
> >>>>>> Josh
> >>>>
> >>>>
> >>
> >>
>
>
Re: setting session params in a rest request
Posted by Josh Schlesser <jo...@spoutable.com>.
Thanks Venki,
Let me know how I can help further.
> On Feb 2, 2016, at 10:23 PM, Venki Korukanti <ve...@gmail.com> wrote:
>
> I think you made valid points. It makes sense to have session less REST
> calls in both auth enabled and disabled cases.
>
> In case of auth enabled:
> 1) Session-less calls can be authenticated using Basic auth (this was
> already asked on mailing list sometime back) as a start and move onto token
> based auth later. These requests usually come from non-browsers. The only
> issue is setting options before the query. For this we can implement your
> suggestion of enhancing the query REST API to accept the options.
> 2) Session-based call using form auth for browser based access. If we
> enhance the UI to enter options in the query form, we don't need any
> session on server actually.
>
> I will get a fix ASAP to remove the sessions in anonymous calls as they the
> session are not reused in non-browser cases.
>
> Thanks
> Venki
>
> On Tue, Feb 2, 2016 at 9:20 PM, Josh Schlesser <jo...@spoutable.com> wrote:
>
>> No, it wasn’t logging out, it was just stopping, obviously that caused
>> dangling sessions for the authenticated scenario.
>>
>> I don’t think that a short timeout for anonymous sessions is a good way to
>> go for anonymous api calls. Session management isn’t what anybody would
>> expect when using a REST api that is anonymous in a server to server
>> context. I would expect to use a token for authorization for a server to
>> server REST api as well. I’m not saying that is what it should be here,
>> but that is what my general expectation is based on using other apis. In
>> the case of browser to server REST apis, I have run into authentication for
>> a browser session and subsequent REST calls leaning on a browser cookie for
>> persistent authentication.
>>
>> Removing sessions for anonymous calls seems like the right path and
>> possibly easy and I think would be the expected behavior from most
>> developers. I would advocate for sessionless and token authenticated REST
>> apis for when using authentication for the server to server case and cookie
>> based with a session for the browser to server scenario, but its really the
>> browser that has a session, not the api per se, its just piggybacking on a
>> regular authenticated web session for the REST api calls.
>>
>> This would actually leave me in a quandary for what I am trying to do
>> which is set a session configuration option ’store.format', but I cant
>> think of any reason that those types of settings shouldn’t just be set on a
>> per request basis for a REST api. In a server to server context for a rest
>> api, keeping it sessionless means you could front a cluster of drillbits
>> with a load balancer and not worry about dying nodes and sticky sessions
>> etc...
>>
>> I have to get something up and running quickly right now so im versioning
>> back to 1.4 and just spinning up a separate drillbit that will have the
>> store.format system variable set to ‘json’ . it will be ok for me until a
>> good long term solution arrives in drill.
>>
>> I’ll run the test on short session_max_idle_secs to 30 seconds on
>> 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation
>> problem, but keep in mind that means that users of the web console will
>> have 30 seconds between pages or they have to authenticate again, which
>> will probably be very annoying. It doesnt seem like a good long term
>> solution either.
>>
>> How do you think all of this should work? I look forward to staying
>> involved.
>>
>> Cheers,
>> Josh
>>
>>> On Feb 2, 2016, at 4:40 PM, Venki Korukanti <ve...@gmail.com>
>> wrote:
>>>
>>> When auth is *enabled*, is the worker process logging out after queries
>> are
>>> done? When auth is *disabled* can you set session_max_idle_secs in
>>> drill.exec.http block in drill-override.conf to something like 30 (secs)
>>> and try? This way anonymous sessions are closed quickly and not kept for
>>> 1hr (default value). I think we may need to avoid creating sessions in
>>> anonymous mode (when auth is disabled).
>>>
>>> Thanks
>>> Venki
>>>
>>> On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <jo...@spoutable.com>
>> wrote:
>>>
>>>> I have a background worker process (on a server, not a browser) that
>> kicks
>>>> off every minute or so and issues some queries sequentially to the rest
>>>> query endpoint. In 1.4 with no authentication this worked fine except
>>>> that in 1 instance I need to issue a CTAS query with a different format
>>>> (json).
>>>>
>>>> I upgraded to 1.5-SNAPSHOT commit
>> bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
>>>>
>>>> Since the upgrade I am getting a resource starvation problem with or
>>>> without authentication
>>>> The drillbit process stays up for a an hour or less and then becomes
>>>> unresponsive and eats up the cpu.
>>>>
>>>> It is definitely a resource starvation issue, not sure if its a resource
>>>> leak.
>>>> Below is a stack trace.
>>>> Also when i lsof on the pid there are a lot (more than a thousand) of
>>>> files like this listed which are used by NIO selectors. so it smells
>> like
>>>> a resource leak.
>>>>
>>>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>>>> java 2931 root 288u 0000 0,11 0 7705
>>>> anon_inode
>>>>
>>>> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
>>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
>>>> java.lang.IllegalStateException: failed to create a child event loop
>>>> at
>>>>
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
>>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>>>> at
>>>>
>> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
>>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>>>> at
>>>>
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
>>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
>>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
>>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
>>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
>>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
>>>> open files
>>>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
>>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>>>> ... 25 common frames omitted
>>>> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
>>>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
>>>> java.lang.IllegalStateException: failed to create a child event loop
>>>> at
>>>>
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
>>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>>>> at
>>>>
>> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
>>>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>>>> at
>>>>
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
>>>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
>>>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
>>>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>>>> at
>>>>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
>>>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
>>>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
>>>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
>>>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at
>>>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>>>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
>>>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
>>>> open files
>>>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
>>>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>>>> at
>>>>
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
>>>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>>>> ... 25 common frames omitted
>>>>
>>>>
>>>>
>>>>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <venki.korukanti@gmail.com
>>>
>>>> wrote:
>>>>>
>>>>> Currently we keep the DrillClient per session. All the state is in
>> Server
>>>>> and DrillClient is the reference to reuse the state. DrillClient is
>>>>> automatically closed when the session expires (default value is 1hr
>> after
>>>>> the last activity on session) or user explicitly logs out. I am trying
>> to
>>>>> understand if there is a resource leak. Do you have too many sessions
>>>> open
>>>>> when the system load is max or just few sessions but you have already
>> ran
>>>>> many queries using the existing sessions? If it is the former it is
>>>>> understandable to have per connection per session life. Also are the
>>>>> resources not freeing up after logout?
>>>>>
>>>>> If you need to have multiple simultaneous sessions, it is better to
>>>> connect
>>>>> to different Drillbits (may be in a round-robin fashion) than always
>>>>> connecting to a single Drillbit.
>>>>>
>>>>> Thanks
>>>>> Venki
>>>>>
>>>>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com
>>>> <ma...@spoutable.com>> wrote:
>>>>>
>>>>>> First: Im a total newb at contributing to apache projects so please
>>>> excuse
>>>>>> any indiscretions, feel free to give comments on style or whatever, i
>>>> take
>>>>>> feedback well. Thick skin too.
>>>>>>
>>>>>>
>>>>>> Ill give some background next and then a proposal.
>>>>>>
>>>>>> Background:
>>>>>> I recently changed over to using authentication in the 1.5 snapshot
>>>>>> because I need to have a session via the REST api so that I can set
>> the
>>>>>> session storage options in an initial query for a subsequent CTAS
>> query.
>>>>>> Previously all rest calls seemed to be completely independent.
>>>>>>
>>>>>> Since the change I have started seeing ‘too many files open’ errors in
>>>> my
>>>>>> drillbit.log and the drillbit java process becomes effectively hung
>>>> waiting
>>>>>> for open file descriptor slots. When running the top command the
>>>> machine
>>>>>> is running at max load due to the drillbit process and the drillbit
>>>> becomes
>>>>>> effectively unresponsive, even the simple pages in the web console
>> don’t
>>>>>> respond. Investigating further it seems that there might be a file
>>>> kept
>>>>>> open per session by the drillbit process for the life of the session.
>>>> I
>>>>>> used the lsof unix command on the drillbit process and found a lot of
>>>> unix
>>>>>> pipes. Looking at the code it looks like these pipes could be for the
>>>>>> communication between the web process and the rpc server, with one
>> being
>>>>>> allocated per session. I haven’t validated this, its just a guess
>> after
>>>>>> scanning the code. I had 1.4 running without this requirement and
>>>> without
>>>>>> ever seeing the error. It seems without authentication the number of
>>>> open
>>>>>> files is a non-issue for me, possibly due to sessions.
>>>>>>
>>>>>> I'm wondering if my guess about what is causing the ‘too many open
>>>> files’
>>>>>> error is plausible? Does anybody with a deeper understanding of the
>>>>>> architecture have any comments on this?
>>>>>>
>>>>>> Proposal:
>>>>>> Assuming sessions are the issue, I am making some changes to my rest
>>>>>> client so that sessions are more effectively used and I can up the
>>>> ulimit
>>>>>> for the drillbit process for the linux user in hopes of mitigating
>>>> this. I
>>>>>> am effectively creating a rest client based session pool that resets
>>>>>> session variables to defaults when the session gets reused.
>> However,
>>>> it
>>>>>> seems hacky.
>>>>>>
>>>>>> Below is an idea for getting per request based settings which seems
>> less
>>>>>> hacky in the long term.
>>>>>>
>>>>>> Can I add a new array member to the query.json REST method in a
>>>> backwards
>>>>>> compatible way to set session level parameters in a single request?
>>>>>> Currently a rest request via the api has a body like so:
>>>>>> { “queryType”: “SQL”, “query” : “<drill query>”}
>>>>>>
>>>>>> id like to do the following
>>>>>>
>>>>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
>>>>>> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
>>>>>>
>>>>>> or even
>>>>>>
>>>>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
>>>> [“SET
>>>>>> `option_name` = value”, “SET `option_name1` = value1”,“SET
>>>> `option_name2` =
>>>>>> value2”, “SET `option_name3` = value3”]}
>>>>>>
>>>>>> As far as I can tell drill is essentially stateless between queries
>>>> right
>>>>>> now except for session level system parameters and authentication.
>>>> There
>>>>>> aren’t any in memory temp tables or cursors or variables like PL/SQL
>> or
>>>>>> PSQL or other SQLs that would make it stateful.
>>>>>>
>>>>>> Given the stateless assumption, being able to set session level params
>>>> on
>>>>>> a per request basis would cover all of the cases that I might need.
>> It
>>>>>> looks relatively straight forward to add something to QueryWrapper to
>>>>>> accept an optional query session settings section of the json packet
>> and
>>>>>> execute those ’SET' commands before the final query. This will work
>>>> for
>>>>>> me, as I can run without authentication in an ’secure' backend
>>>> environment
>>>>>> which will remove sessions and hence file descriptors, assuming my
>>>>>> assumptions about file descriptors and sessions are correct.
>>>>>>
>>>>>>
>>>>>> My java is rusty (circa 2003) but some casual googling implies that if
>>>>>> this were added as a 3rd @FormParam to submitQuery in QueryResources
>> it
>>>>>> would be magically be null if it werent present and could easily be
>>>>>> ignored. If its present then an alternative constructor of
>> QueryWrapper
>>>>>> could be called with the extra param and it would be easy to alter its
>>>> run
>>>>>> method to execute the SET commands. There would need to be some error
>>>>>> handling of course if the SET commands were illegal or failed to run
>> for
>>>>>> some reason.
>>>>>>
>>>>>> If this seems reasonable, how do I go about contributing? I looked
>>>>>> through the links in the docs to apache foundation incubator projects
>>>> but
>>>>>> the links to drill were broken :( http://drill.apache.org/team.html
>> <
>>>>>> http://drill.apache.org/team.html <http://drill.apache.org/team.html
>>>>
>>>> I read this
>>>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <
>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> <
>>>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
>>>> and
>>>>>> i have subscribed to the dev mailing list (obvious since you are
>> getting
>>>>>> this). It said to post here before creating a JIRA. Am I missing
>>>>>> anything in my assumptions? Comments? Should I just submit a JIRA
>> and
>>>> a
>>>>>> patch or submit a JIRA and a comment or wait for comments before
>> coding
>>>>>> stuff up as an example?
>>>>>>
>>>>>> Thanks for taking the time to read and respond.
>>>>>>
>>>>>> Josh
>>>>
>>>>
>>
>>
Re: setting session params in a rest request
Posted by Venki Korukanti <ve...@gmail.com>.
I think you made valid points. It makes sense to have session less REST
calls in both auth enabled and disabled cases.
In case of auth enabled:
1) Session-less calls can be authenticated using Basic auth (this was
already asked on mailing list sometime back) as a start and move onto token
based auth later. These requests usually come from non-browsers. The only
issue is setting options before the query. For this we can implement your
suggestion of enhancing the query REST API to accept the options.
2) Session-based call using form auth for browser based access. If we
enhance the UI to enter options in the query form, we don't need any
session on server actually.
I will get a fix ASAP to remove the sessions in anonymous calls as they the
session are not reused in non-browser cases.
Thanks
Venki
On Tue, Feb 2, 2016 at 9:20 PM, Josh Schlesser <jo...@spoutable.com> wrote:
> No, it wasn’t logging out, it was just stopping, obviously that caused
> dangling sessions for the authenticated scenario.
>
> I don’t think that a short timeout for anonymous sessions is a good way to
> go for anonymous api calls. Session management isn’t what anybody would
> expect when using a REST api that is anonymous in a server to server
> context. I would expect to use a token for authorization for a server to
> server REST api as well. I’m not saying that is what it should be here,
> but that is what my general expectation is based on using other apis. In
> the case of browser to server REST apis, I have run into authentication for
> a browser session and subsequent REST calls leaning on a browser cookie for
> persistent authentication.
>
> Removing sessions for anonymous calls seems like the right path and
> possibly easy and I think would be the expected behavior from most
> developers. I would advocate for sessionless and token authenticated REST
> apis for when using authentication for the server to server case and cookie
> based with a session for the browser to server scenario, but its really the
> browser that has a session, not the api per se, its just piggybacking on a
> regular authenticated web session for the REST api calls.
>
> This would actually leave me in a quandary for what I am trying to do
> which is set a session configuration option ’store.format', but I cant
> think of any reason that those types of settings shouldn’t just be set on a
> per request basis for a REST api. In a server to server context for a rest
> api, keeping it sessionless means you could front a cluster of drillbits
> with a load balancer and not worry about dying nodes and sticky sessions
> etc...
>
> I have to get something up and running quickly right now so im versioning
> back to 1.4 and just spinning up a separate drillbit that will have the
> store.format system variable set to ‘json’ . it will be ok for me until a
> good long term solution arrives in drill.
>
> I’ll run the test on short session_max_idle_secs to 30 seconds on
> 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation
> problem, but keep in mind that means that users of the web console will
> have 30 seconds between pages or they have to authenticate again, which
> will probably be very annoying. It doesnt seem like a good long term
> solution either.
>
> How do you think all of this should work? I look forward to staying
> involved.
>
> Cheers,
> Josh
>
> > On Feb 2, 2016, at 4:40 PM, Venki Korukanti <ve...@gmail.com>
> wrote:
> >
> > When auth is *enabled*, is the worker process logging out after queries
> are
> > done? When auth is *disabled* can you set session_max_idle_secs in
> > drill.exec.http block in drill-override.conf to something like 30 (secs)
> > and try? This way anonymous sessions are closed quickly and not kept for
> > 1hr (default value). I think we may need to avoid creating sessions in
> > anonymous mode (when auth is disabled).
> >
> > Thanks
> > Venki
> >
> > On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <jo...@spoutable.com>
> wrote:
> >
> >> I have a background worker process (on a server, not a browser) that
> kicks
> >> off every minute or so and issues some queries sequentially to the rest
> >> query endpoint. In 1.4 with no authentication this worked fine except
> >> that in 1 instance I need to issue a CTAS query with a different format
> >> (json).
> >>
> >> I upgraded to 1.5-SNAPSHOT commit
> bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
> >>
> >> Since the upgrade I am getting a resource starvation problem with or
> >> without authentication
> >> The drillbit process stays up for a an hour or less and then becomes
> >> unresponsive and eats up the cpu.
> >>
> >> It is definitely a resource starvation issue, not sure if its a resource
> >> leak.
> >> Below is a stack trace.
> >> Also when i lsof on the pid there are a lot (more than a thousand) of
> >> files like this listed which are used by NIO selectors. so it smells
> like
> >> a resource leak.
> >>
> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> >> java 2931 root 288u 0000 0,11 0 7705
> >> anon_inode
> >>
> >> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
> >> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >> java.lang.IllegalStateException: failed to create a child event loop
> >> at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >> at
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >> at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
> >> open files
> >> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >> ... 25 common frames omitted
> >> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
> >> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> >> java.lang.IllegalStateException: failed to create a child event loop
> >> at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >> at
> >>
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> >> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> >> at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> >> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> >> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> >> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> >> at
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> >> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> >> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> >> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> >> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at
> >>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> >> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> >> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> >> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
> >> open files
> >> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> >> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> >> at
> >>
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> >> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> >> ... 25 common frames omitted
> >>
> >>
> >>
> >>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <venki.korukanti@gmail.com
> >
> >> wrote:
> >>>
> >>> Currently we keep the DrillClient per session. All the state is in
> Server
> >>> and DrillClient is the reference to reuse the state. DrillClient is
> >>> automatically closed when the session expires (default value is 1hr
> after
> >>> the last activity on session) or user explicitly logs out. I am trying
> to
> >>> understand if there is a resource leak. Do you have too many sessions
> >> open
> >>> when the system load is max or just few sessions but you have already
> ran
> >>> many queries using the existing sessions? If it is the former it is
> >>> understandable to have per connection per session life. Also are the
> >>> resources not freeing up after logout?
> >>>
> >>> If you need to have multiple simultaneous sessions, it is better to
> >> connect
> >>> to different Drillbits (may be in a round-robin fashion) than always
> >>> connecting to a single Drillbit.
> >>>
> >>> Thanks
> >>> Venki
> >>>
> >>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com
> >> <ma...@spoutable.com>> wrote:
> >>>
> >>>> First: Im a total newb at contributing to apache projects so please
> >> excuse
> >>>> any indiscretions, feel free to give comments on style or whatever, i
> >> take
> >>>> feedback well. Thick skin too.
> >>>>
> >>>>
> >>>> Ill give some background next and then a proposal.
> >>>>
> >>>> Background:
> >>>> I recently changed over to using authentication in the 1.5 snapshot
> >>>> because I need to have a session via the REST api so that I can set
> the
> >>>> session storage options in an initial query for a subsequent CTAS
> query.
> >>>> Previously all rest calls seemed to be completely independent.
> >>>>
> >>>> Since the change I have started seeing ‘too many files open’ errors in
> >> my
> >>>> drillbit.log and the drillbit java process becomes effectively hung
> >> waiting
> >>>> for open file descriptor slots. When running the top command the
> >> machine
> >>>> is running at max load due to the drillbit process and the drillbit
> >> becomes
> >>>> effectively unresponsive, even the simple pages in the web console
> don’t
> >>>> respond. Investigating further it seems that there might be a file
> >> kept
> >>>> open per session by the drillbit process for the life of the session.
> >> I
> >>>> used the lsof unix command on the drillbit process and found a lot of
> >> unix
> >>>> pipes. Looking at the code it looks like these pipes could be for the
> >>>> communication between the web process and the rpc server, with one
> being
> >>>> allocated per session. I haven’t validated this, its just a guess
> after
> >>>> scanning the code. I had 1.4 running without this requirement and
> >> without
> >>>> ever seeing the error. It seems without authentication the number of
> >> open
> >>>> files is a non-issue for me, possibly due to sessions.
> >>>>
> >>>> I'm wondering if my guess about what is causing the ‘too many open
> >> files’
> >>>> error is plausible? Does anybody with a deeper understanding of the
> >>>> architecture have any comments on this?
> >>>>
> >>>> Proposal:
> >>>> Assuming sessions are the issue, I am making some changes to my rest
> >>>> client so that sessions are more effectively used and I can up the
> >> ulimit
> >>>> for the drillbit process for the linux user in hopes of mitigating
> >> this. I
> >>>> am effectively creating a rest client based session pool that resets
> >>>> session variables to defaults when the session gets reused.
> However,
> >> it
> >>>> seems hacky.
> >>>>
> >>>> Below is an idea for getting per request based settings which seems
> less
> >>>> hacky in the long term.
> >>>>
> >>>> Can I add a new array member to the query.json REST method in a
> >> backwards
> >>>> compatible way to set session level parameters in a single request?
> >>>> Currently a rest request via the api has a body like so:
> >>>> { “queryType”: “SQL”, “query” : “<drill query>”}
> >>>>
> >>>> id like to do the following
> >>>>
> >>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> >>>> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
> >>>>
> >>>> or even
> >>>>
> >>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> >> [“SET
> >>>> `option_name` = value”, “SET `option_name1` = value1”,“SET
> >> `option_name2` =
> >>>> value2”, “SET `option_name3` = value3”]}
> >>>>
> >>>> As far as I can tell drill is essentially stateless between queries
> >> right
> >>>> now except for session level system parameters and authentication.
> >> There
> >>>> aren’t any in memory temp tables or cursors or variables like PL/SQL
> or
> >>>> PSQL or other SQLs that would make it stateful.
> >>>>
> >>>> Given the stateless assumption, being able to set session level params
> >> on
> >>>> a per request basis would cover all of the cases that I might need.
> It
> >>>> looks relatively straight forward to add something to QueryWrapper to
> >>>> accept an optional query session settings section of the json packet
> and
> >>>> execute those ’SET' commands before the final query. This will work
> >> for
> >>>> me, as I can run without authentication in an ’secure' backend
> >> environment
> >>>> which will remove sessions and hence file descriptors, assuming my
> >>>> assumptions about file descriptors and sessions are correct.
> >>>>
> >>>>
> >>>> My java is rusty (circa 2003) but some casual googling implies that if
> >>>> this were added as a 3rd @FormParam to submitQuery in QueryResources
> it
> >>>> would be magically be null if it werent present and could easily be
> >>>> ignored. If its present then an alternative constructor of
> QueryWrapper
> >>>> could be called with the extra param and it would be easy to alter its
> >> run
> >>>> method to execute the SET commands. There would need to be some error
> >>>> handling of course if the SET commands were illegal or failed to run
> for
> >>>> some reason.
> >>>>
> >>>> If this seems reasonable, how do I go about contributing? I looked
> >>>> through the links in the docs to apache foundation incubator projects
> >> but
> >>>> the links to drill were broken :( http://drill.apache.org/team.html
> <
> >>>> http://drill.apache.org/team.html <http://drill.apache.org/team.html
> >>
> >> I read this
> >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <
> >> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> <
> >>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
> >> and
> >>>> i have subscribed to the dev mailing list (obvious since you are
> getting
> >>>> this). It said to post here before creating a JIRA. Am I missing
> >>>> anything in my assumptions? Comments? Should I just submit a JIRA
> and
> >> a
> >>>> patch or submit a JIRA and a comment or wait for comments before
> coding
> >>>> stuff up as an example?
> >>>>
> >>>> Thanks for taking the time to read and respond.
> >>>>
> >>>> Josh
> >>
> >>
>
>
Re: setting session params in a rest request
Posted by Josh Schlesser <jo...@spoutable.com>.
No, it wasn’t logging out, it was just stopping, obviously that caused dangling sessions for the authenticated scenario.
I don’t think that a short timeout for anonymous sessions is a good way to go for anonymous api calls. Session management isn’t what anybody would expect when using a REST api that is anonymous in a server to server context. I would expect to use a token for authorization for a server to server REST api as well. I’m not saying that is what it should be here, but that is what my general expectation is based on using other apis. In the case of browser to server REST apis, I have run into authentication for a browser session and subsequent REST calls leaning on a browser cookie for persistent authentication.
Removing sessions for anonymous calls seems like the right path and possibly easy and I think would be the expected behavior from most developers. I would advocate for sessionless and token authenticated REST apis for when using authentication for the server to server case and cookie based with a session for the browser to server scenario, but its really the browser that has a session, not the api per se, its just piggybacking on a regular authenticated web session for the REST api calls.
This would actually leave me in a quandary for what I am trying to do which is set a session configuration option ’store.format', but I cant think of any reason that those types of settings shouldn’t just be set on a per request basis for a REST api. In a server to server context for a rest api, keeping it sessionless means you could front a cluster of drillbits with a load balancer and not worry about dying nodes and sticky sessions etc...
I have to get something up and running quickly right now so im versioning back to 1.4 and just spinning up a separate drillbit that will have the store.format system variable set to ‘json’ . it will be ok for me until a good long term solution arrives in drill.
I’ll run the test on short session_max_idle_secs to 30 seconds on 1.5.0-SNAPSHOT to see if that gets rid of the file handle starvation problem, but keep in mind that means that users of the web console will have 30 seconds between pages or they have to authenticate again, which will probably be very annoying. It doesnt seem like a good long term solution either.
How do you think all of this should work? I look forward to staying involved.
Cheers,
Josh
> On Feb 2, 2016, at 4:40 PM, Venki Korukanti <ve...@gmail.com> wrote:
>
> When auth is *enabled*, is the worker process logging out after queries are
> done? When auth is *disabled* can you set session_max_idle_secs in
> drill.exec.http block in drill-override.conf to something like 30 (secs)
> and try? This way anonymous sessions are closed quickly and not kept for
> 1hr (default value). I think we may need to avoid creating sessions in
> anonymous mode (when auth is disabled).
>
> Thanks
> Venki
>
> On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <jo...@spoutable.com> wrote:
>
>> I have a background worker process (on a server, not a browser) that kicks
>> off every minute or so and issues some queries sequentially to the rest
>> query endpoint. In 1.4 with no authentication this worked fine except
>> that in 1 instance I need to issue a CTAS query with a different format
>> (json).
>>
>> I upgraded to 1.5-SNAPSHOT commit bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
>>
>> Since the upgrade I am getting a resource starvation problem with or
>> without authentication
>> The drillbit process stays up for a an hour or less and then becomes
>> unresponsive and eats up the cpu.
>>
>> It is definitely a resource starvation issue, not sure if its a resource
>> leak.
>> Below is a stack trace.
>> Also when i lsof on the pid there are a lot (more than a thousand) of
>> files like this listed which are used by NIO selectors. so it smells like
>> a resource leak.
>>
>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
>> java 2931 root 288u 0000 0,11 0 7705
>> anon_inode
>>
>> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
>> java.lang.IllegalStateException: failed to create a child event loop
>> at
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>> at
>> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>> at
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
>> open files
>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>> ... 25 common frames omitted
>> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
>> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
>> java.lang.IllegalStateException: failed to create a child event loop
>> at
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>> at
>> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
>> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
>> at
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
>> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
>> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
>> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
>> at
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
>> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
>> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at org.eclipse.jetty.server.Server.handle(Server.java:462)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
>> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
>> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
>> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
>> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
>> open files
>> at io.netty.channel.epoll.Native.epollCreate(Native Method)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
>> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
>> at
>> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
>> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
>> ... 25 common frames omitted
>>
>>
>>
>>> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <ve...@gmail.com>
>> wrote:
>>>
>>> Currently we keep the DrillClient per session. All the state is in Server
>>> and DrillClient is the reference to reuse the state. DrillClient is
>>> automatically closed when the session expires (default value is 1hr after
>>> the last activity on session) or user explicitly logs out. I am trying to
>>> understand if there is a resource leak. Do you have too many sessions
>> open
>>> when the system load is max or just few sessions but you have already ran
>>> many queries using the existing sessions? If it is the former it is
>>> understandable to have per connection per session life. Also are the
>>> resources not freeing up after logout?
>>>
>>> If you need to have multiple simultaneous sessions, it is better to
>> connect
>>> to different Drillbits (may be in a round-robin fashion) than always
>>> connecting to a single Drillbit.
>>>
>>> Thanks
>>> Venki
>>>
>>> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com
>> <ma...@spoutable.com>> wrote:
>>>
>>>> First: Im a total newb at contributing to apache projects so please
>> excuse
>>>> any indiscretions, feel free to give comments on style or whatever, i
>> take
>>>> feedback well. Thick skin too.
>>>>
>>>>
>>>> Ill give some background next and then a proposal.
>>>>
>>>> Background:
>>>> I recently changed over to using authentication in the 1.5 snapshot
>>>> because I need to have a session via the REST api so that I can set the
>>>> session storage options in an initial query for a subsequent CTAS query.
>>>> Previously all rest calls seemed to be completely independent.
>>>>
>>>> Since the change I have started seeing ‘too many files open’ errors in
>> my
>>>> drillbit.log and the drillbit java process becomes effectively hung
>> waiting
>>>> for open file descriptor slots. When running the top command the
>> machine
>>>> is running at max load due to the drillbit process and the drillbit
>> becomes
>>>> effectively unresponsive, even the simple pages in the web console don’t
>>>> respond. Investigating further it seems that there might be a file
>> kept
>>>> open per session by the drillbit process for the life of the session.
>> I
>>>> used the lsof unix command on the drillbit process and found a lot of
>> unix
>>>> pipes. Looking at the code it looks like these pipes could be for the
>>>> communication between the web process and the rpc server, with one being
>>>> allocated per session. I haven’t validated this, its just a guess after
>>>> scanning the code. I had 1.4 running without this requirement and
>> without
>>>> ever seeing the error. It seems without authentication the number of
>> open
>>>> files is a non-issue for me, possibly due to sessions.
>>>>
>>>> I'm wondering if my guess about what is causing the ‘too many open
>> files’
>>>> error is plausible? Does anybody with a deeper understanding of the
>>>> architecture have any comments on this?
>>>>
>>>> Proposal:
>>>> Assuming sessions are the issue, I am making some changes to my rest
>>>> client so that sessions are more effectively used and I can up the
>> ulimit
>>>> for the drillbit process for the linux user in hopes of mitigating
>> this. I
>>>> am effectively creating a rest client based session pool that resets
>>>> session variables to defaults when the session gets reused. However,
>> it
>>>> seems hacky.
>>>>
>>>> Below is an idea for getting per request based settings which seems less
>>>> hacky in the long term.
>>>>
>>>> Can I add a new array member to the query.json REST method in a
>> backwards
>>>> compatible way to set session level parameters in a single request?
>>>> Currently a rest request via the api has a body like so:
>>>> { “queryType”: “SQL”, “query” : “<drill query>”}
>>>>
>>>> id like to do the following
>>>>
>>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
>>>> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
>>>>
>>>> or even
>>>>
>>>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
>> [“SET
>>>> `option_name` = value”, “SET `option_name1` = value1”,“SET
>> `option_name2` =
>>>> value2”, “SET `option_name3` = value3”]}
>>>>
>>>> As far as I can tell drill is essentially stateless between queries
>> right
>>>> now except for session level system parameters and authentication.
>> There
>>>> aren’t any in memory temp tables or cursors or variables like PL/SQL or
>>>> PSQL or other SQLs that would make it stateful.
>>>>
>>>> Given the stateless assumption, being able to set session level params
>> on
>>>> a per request basis would cover all of the cases that I might need. It
>>>> looks relatively straight forward to add something to QueryWrapper to
>>>> accept an optional query session settings section of the json packet and
>>>> execute those ’SET' commands before the final query. This will work
>> for
>>>> me, as I can run without authentication in an ’secure' backend
>> environment
>>>> which will remove sessions and hence file descriptors, assuming my
>>>> assumptions about file descriptors and sessions are correct.
>>>>
>>>>
>>>> My java is rusty (circa 2003) but some casual googling implies that if
>>>> this were added as a 3rd @FormParam to submitQuery in QueryResources it
>>>> would be magically be null if it werent present and could easily be
>>>> ignored. If its present then an alternative constructor of QueryWrapper
>>>> could be called with the extra param and it would be easy to alter its
>> run
>>>> method to execute the SET commands. There would need to be some error
>>>> handling of course if the SET commands were illegal or failed to run for
>>>> some reason.
>>>>
>>>> If this seems reasonable, how do I go about contributing? I looked
>>>> through the links in the docs to apache foundation incubator projects
>> but
>>>> the links to drill were broken :( http://drill.apache.org/team.html <
>>>> http://drill.apache.org/team.html <http://drill.apache.org/team.html>>
>> I read this
>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <
>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> <
>>>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
>> and
>>>> i have subscribed to the dev mailing list (obvious since you are getting
>>>> this). It said to post here before creating a JIRA. Am I missing
>>>> anything in my assumptions? Comments? Should I just submit a JIRA and
>> a
>>>> patch or submit a JIRA and a comment or wait for comments before coding
>>>> stuff up as an example?
>>>>
>>>> Thanks for taking the time to read and respond.
>>>>
>>>> Josh
>>
>>
Re: setting session params in a rest request
Posted by Venki Korukanti <ve...@gmail.com>.
When auth is *enabled*, is the worker process logging out after queries are
done? When auth is *disabled* can you set session_max_idle_secs in
drill.exec.http block in drill-override.conf to something like 30 (secs)
and try? This way anonymous sessions are closed quickly and not kept for
1hr (default value). I think we may need to avoid creating sessions in
anonymous mode (when auth is disabled).
Thanks
Venki
On Tue, Feb 2, 2016 at 4:02 PM, Josh Schlesser <jo...@spoutable.com> wrote:
> I have a background worker process (on a server, not a browser) that kicks
> off every minute or so and issues some queries sequentially to the rest
> query endpoint. In 1.4 with no authentication this worked fine except
> that in 1 instance I need to issue a CTAS query with a different format
> (json).
>
> I upgraded to 1.5-SNAPSHOT commit bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
>
> Since the upgrade I am getting a resource starvation problem with or
> without authentication
> The drillbit process stays up for a an hour or less and then becomes
> unresponsive and eats up the cpu.
>
> It is definitely a resource starvation issue, not sure if its a resource
> leak.
> Below is a stack trace.
> Also when i lsof on the pid there are a lot (more than a thousand) of
> files like this listed which are used by NIO selectors. so it smells like
> a resource leak.
>
> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
> java 2931 root 288u 0000 0,11 0 7705
> anon_inode
>
> 2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR
> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> java.lang.IllegalStateException: failed to create a child event loop
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
> open files
> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> ... 25 common frames omitted
> 2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR
> o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
> java.lang.IllegalStateException: failed to create a child event loop
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68)
> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49)
> ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
> at
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73)
> ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47)
> ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71)
> [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503)
> [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
> [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at org.eclipse.jetty.server.Server.handle(Server.java:462)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
> [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
> [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
> [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
> Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many
> open files
> at io.netty.channel.epoll.Native.epollCreate(Native Method)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76)
> ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
> at
> io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64)
> ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
> ... 25 common frames omitted
>
>
>
> > On Feb 2, 2016, at 7:40 AM, Venki Korukanti <ve...@gmail.com>
> wrote:
> >
> > Currently we keep the DrillClient per session. All the state is in Server
> > and DrillClient is the reference to reuse the state. DrillClient is
> > automatically closed when the session expires (default value is 1hr after
> > the last activity on session) or user explicitly logs out. I am trying to
> > understand if there is a resource leak. Do you have too many sessions
> open
> > when the system load is max or just few sessions but you have already ran
> > many queries using the existing sessions? If it is the former it is
> > understandable to have per connection per session life. Also are the
> > resources not freeing up after logout?
> >
> > If you need to have multiple simultaneous sessions, it is better to
> connect
> > to different Drillbits (may be in a round-robin fashion) than always
> > connecting to a single Drillbit.
> >
> > Thanks
> > Venki
> >
> > On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com
> <ma...@spoutable.com>> wrote:
> >
> >> First: Im a total newb at contributing to apache projects so please
> excuse
> >> any indiscretions, feel free to give comments on style or whatever, i
> take
> >> feedback well. Thick skin too.
> >>
> >>
> >> Ill give some background next and then a proposal.
> >>
> >> Background:
> >> I recently changed over to using authentication in the 1.5 snapshot
> >> because I need to have a session via the REST api so that I can set the
> >> session storage options in an initial query for a subsequent CTAS query.
> >> Previously all rest calls seemed to be completely independent.
> >>
> >> Since the change I have started seeing ‘too many files open’ errors in
> my
> >> drillbit.log and the drillbit java process becomes effectively hung
> waiting
> >> for open file descriptor slots. When running the top command the
> machine
> >> is running at max load due to the drillbit process and the drillbit
> becomes
> >> effectively unresponsive, even the simple pages in the web console don’t
> >> respond. Investigating further it seems that there might be a file
> kept
> >> open per session by the drillbit process for the life of the session.
> I
> >> used the lsof unix command on the drillbit process and found a lot of
> unix
> >> pipes. Looking at the code it looks like these pipes could be for the
> >> communication between the web process and the rpc server, with one being
> >> allocated per session. I haven’t validated this, its just a guess after
> >> scanning the code. I had 1.4 running without this requirement and
> without
> >> ever seeing the error. It seems without authentication the number of
> open
> >> files is a non-issue for me, possibly due to sessions.
> >>
> >> I'm wondering if my guess about what is causing the ‘too many open
> files’
> >> error is plausible? Does anybody with a deeper understanding of the
> >> architecture have any comments on this?
> >>
> >> Proposal:
> >> Assuming sessions are the issue, I am making some changes to my rest
> >> client so that sessions are more effectively used and I can up the
> ulimit
> >> for the drillbit process for the linux user in hopes of mitigating
> this. I
> >> am effectively creating a rest client based session pool that resets
> >> session variables to defaults when the session gets reused. However,
> it
> >> seems hacky.
> >>
> >> Below is an idea for getting per request based settings which seems less
> >> hacky in the long term.
> >>
> >> Can I add a new array member to the query.json REST method in a
> backwards
> >> compatible way to set session level parameters in a single request?
> >> Currently a rest request via the api has a body like so:
> >> { “queryType”: “SQL”, “query” : “<drill query>”}
> >>
> >> id like to do the following
> >>
> >> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> >> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
> >>
> >> or even
> >>
> >> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> [“SET
> >> `option_name` = value”, “SET `option_name1` = value1”,“SET
> `option_name2` =
> >> value2”, “SET `option_name3` = value3”]}
> >>
> >> As far as I can tell drill is essentially stateless between queries
> right
> >> now except for session level system parameters and authentication.
> There
> >> aren’t any in memory temp tables or cursors or variables like PL/SQL or
> >> PSQL or other SQLs that would make it stateful.
> >>
> >> Given the stateless assumption, being able to set session level params
> on
> >> a per request basis would cover all of the cases that I might need. It
> >> looks relatively straight forward to add something to QueryWrapper to
> >> accept an optional query session settings section of the json packet and
> >> execute those ’SET' commands before the final query. This will work
> for
> >> me, as I can run without authentication in an ’secure' backend
> environment
> >> which will remove sessions and hence file descriptors, assuming my
> >> assumptions about file descriptors and sessions are correct.
> >>
> >>
> >> My java is rusty (circa 2003) but some casual googling implies that if
> >> this were added as a 3rd @FormParam to submitQuery in QueryResources it
> >> would be magically be null if it werent present and could easily be
> >> ignored. If its present then an alternative constructor of QueryWrapper
> >> could be called with the extra param and it would be easy to alter its
> run
> >> method to execute the SET commands. There would need to be some error
> >> handling of course if the SET commands were illegal or failed to run for
> >> some reason.
> >>
> >> If this seems reasonable, how do I go about contributing? I looked
> >> through the links in the docs to apache foundation incubator projects
> but
> >> the links to drill were broken :( http://drill.apache.org/team.html <
> >> http://drill.apache.org/team.html <http://drill.apache.org/team.html>>
> I read this
> >> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <
> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> <
> >> http://drill.apache.org/docs/apache-drill-contribution-guidelines/>
> and
> >> i have subscribed to the dev mailing list (obvious since you are getting
> >> this). It said to post here before creating a JIRA. Am I missing
> >> anything in my assumptions? Comments? Should I just submit a JIRA and
> a
> >> patch or submit a JIRA and a comment or wait for comments before coding
> >> stuff up as an example?
> >>
> >> Thanks for taking the time to read and respond.
> >>
> >> Josh
>
>
Re: setting session params in a rest request
Posted by Josh Schlesser <jo...@spoutable.com>.
I have a background worker process (on a server, not a browser) that kicks off every minute or so and issues some queries sequentially to the rest query endpoint. In 1.4 with no authentication this worked fine except that in 1 instance I need to issue a CTAS query with a different format (json).
I upgraded to 1.5-SNAPSHOT commit bb3fc15216d9cab804fc9a6f0e5bd34597dd4394
Since the upgrade I am getting a resource starvation problem with or without authentication
The drillbit process stays up for a an hour or less and then becomes unresponsive and eats up the cpu.
It is definitely a resource starvation issue, not sure if its a resource leak.
Below is a stack trace.
Also when i lsof on the pid there are a lot (more than a thousand) of files like this listed which are used by NIO selectors. so it smells like a resource leak.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 2931 root 288u 0000 0,11 0 7705 anon_inode
2016-02-02 21:56:26,520 [qtp1250890858-11590] ERROR o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73) ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71) [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503) [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.Server.handle(Server.java:462) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many open files
at io.netty.channel.epoll.Native.epollCreate(Native Method) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
... 25 common frames omitted
2016-02-02 21:56:30,130 [qtp1250890858-11591] ERROR o.a.d.e.s.r.a.AnonymousLoginService - Login failed.
java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:68) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:49) ~[netty-transport-4.0.27.Final.jar:4.0.27.Final]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:61) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:49) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at org.apache.drill.exec.rpc.TransportCheck.createEventLoopGroup(TransportCheck.java:73) ~[drill-rpc-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.createEventLoop(DrillClient.java:239) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:220) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.client.DrillClient.connect(DrillClient.java:178) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.server.rest.auth.AbstractDrillLoginService.createDrillClient(AbstractDrillLoginService.java:56) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.server.rest.auth.AnonymousLoginService.login(AnonymousLoginService.java:47) ~[drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.apache.drill.exec.server.rest.auth.AnonymousAuthenticator.validateRequest(AnonymousAuthenticator.java:71) [drill-java-exec-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:503) [jetty-security-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) [jetty-servlet-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.Server.handle(Server.java:462) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) [jetty-server-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) [jetty-io-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) [jetty-util-9.1.5.v20140505.jar:9.1.5.v20140505]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
Caused by: java.lang.RuntimeException: epoll_create1() failed: Too many open files
at io.netty.channel.epoll.Native.epollCreate(Native Method) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:74) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:76) ~[netty-transport-native-epoll-4.0.27.Final-linux-x86_64.jar:na]
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:64) ~[netty-common-4.0.27.Final.jar:4.0.27.Final]
... 25 common frames omitted
> On Feb 2, 2016, at 7:40 AM, Venki Korukanti <ve...@gmail.com> wrote:
>
> Currently we keep the DrillClient per session. All the state is in Server
> and DrillClient is the reference to reuse the state. DrillClient is
> automatically closed when the session expires (default value is 1hr after
> the last activity on session) or user explicitly logs out. I am trying to
> understand if there is a resource leak. Do you have too many sessions open
> when the system load is max or just few sessions but you have already ran
> many queries using the existing sessions? If it is the former it is
> understandable to have per connection per session life. Also are the
> resources not freeing up after logout?
>
> If you need to have multiple simultaneous sessions, it is better to connect
> to different Drillbits (may be in a round-robin fashion) than always
> connecting to a single Drillbit.
>
> Thanks
> Venki
>
> On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <josh@spoutable.com <ma...@spoutable.com>> wrote:
>
>> First: Im a total newb at contributing to apache projects so please excuse
>> any indiscretions, feel free to give comments on style or whatever, i take
>> feedback well. Thick skin too.
>>
>>
>> Ill give some background next and then a proposal.
>>
>> Background:
>> I recently changed over to using authentication in the 1.5 snapshot
>> because I need to have a session via the REST api so that I can set the
>> session storage options in an initial query for a subsequent CTAS query.
>> Previously all rest calls seemed to be completely independent.
>>
>> Since the change I have started seeing ‘too many files open’ errors in my
>> drillbit.log and the drillbit java process becomes effectively hung waiting
>> for open file descriptor slots. When running the top command the machine
>> is running at max load due to the drillbit process and the drillbit becomes
>> effectively unresponsive, even the simple pages in the web console don’t
>> respond. Investigating further it seems that there might be a file kept
>> open per session by the drillbit process for the life of the session. I
>> used the lsof unix command on the drillbit process and found a lot of unix
>> pipes. Looking at the code it looks like these pipes could be for the
>> communication between the web process and the rpc server, with one being
>> allocated per session. I haven’t validated this, its just a guess after
>> scanning the code. I had 1.4 running without this requirement and without
>> ever seeing the error. It seems without authentication the number of open
>> files is a non-issue for me, possibly due to sessions.
>>
>> I'm wondering if my guess about what is causing the ‘too many open files’
>> error is plausible? Does anybody with a deeper understanding of the
>> architecture have any comments on this?
>>
>> Proposal:
>> Assuming sessions are the issue, I am making some changes to my rest
>> client so that sessions are more effectively used and I can up the ulimit
>> for the drillbit process for the linux user in hopes of mitigating this. I
>> am effectively creating a rest client based session pool that resets
>> session variables to defaults when the session gets reused. However, it
>> seems hacky.
>>
>> Below is an idea for getting per request based settings which seems less
>> hacky in the long term.
>>
>> Can I add a new array member to the query.json REST method in a backwards
>> compatible way to set session level parameters in a single request?
>> Currently a rest request via the api has a body like so:
>> { “queryType”: “SQL”, “query” : “<drill query>”}
>>
>> id like to do the following
>>
>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
>> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
>>
>> or even
>>
>> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”: [“SET
>> `option_name` = value”, “SET `option_name1` = value1”,“SET `option_name2` =
>> value2”, “SET `option_name3` = value3”]}
>>
>> As far as I can tell drill is essentially stateless between queries right
>> now except for session level system parameters and authentication. There
>> aren’t any in memory temp tables or cursors or variables like PL/SQL or
>> PSQL or other SQLs that would make it stateful.
>>
>> Given the stateless assumption, being able to set session level params on
>> a per request basis would cover all of the cases that I might need. It
>> looks relatively straight forward to add something to QueryWrapper to
>> accept an optional query session settings section of the json packet and
>> execute those ’SET' commands before the final query. This will work for
>> me, as I can run without authentication in an ’secure' backend environment
>> which will remove sessions and hence file descriptors, assuming my
>> assumptions about file descriptors and sessions are correct.
>>
>>
>> My java is rusty (circa 2003) but some casual googling implies that if
>> this were added as a 3rd @FormParam to submitQuery in QueryResources it
>> would be magically be null if it werent present and could easily be
>> ignored. If its present then an alternative constructor of QueryWrapper
>> could be called with the extra param and it would be easy to alter its run
>> method to execute the SET commands. There would need to be some error
>> handling of course if the SET commands were illegal or failed to run for
>> some reason.
>>
>> If this seems reasonable, how do I go about contributing? I looked
>> through the links in the docs to apache foundation incubator projects but
>> the links to drill were broken :( http://drill.apache.org/team.html <
>> http://drill.apache.org/team.html <http://drill.apache.org/team.html>> I read this
>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <http://drill.apache.org/docs/apache-drill-contribution-guidelines/> <
>> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> and
>> i have subscribed to the dev mailing list (obvious since you are getting
>> this). It said to post here before creating a JIRA. Am I missing
>> anything in my assumptions? Comments? Should I just submit a JIRA and a
>> patch or submit a JIRA and a comment or wait for comments before coding
>> stuff up as an example?
>>
>> Thanks for taking the time to read and respond.
>>
>> Josh
Re: setting session params in a rest request
Posted by Venki Korukanti <ve...@gmail.com>.
Currently we keep the DrillClient per session. All the state is in Server
and DrillClient is the reference to reuse the state. DrillClient is
automatically closed when the session expires (default value is 1hr after
the last activity on session) or user explicitly logs out. I am trying to
understand if there is a resource leak. Do you have too many sessions open
when the system load is max or just few sessions but you have already ran
many queries using the existing sessions? If it is the former it is
understandable to have per connection per session life. Also are the
resources not freeing up after logout?
If you need to have multiple simultaneous sessions, it is better to connect
to different Drillbits (may be in a round-robin fashion) than always
connecting to a single Drillbit.
Thanks
Venki
On Mon, Feb 1, 2016 at 11:51 PM, Josh Schlesser <jo...@spoutable.com> wrote:
> First: Im a total newb at contributing to apache projects so please excuse
> any indiscretions, feel free to give comments on style or whatever, i take
> feedback well. Thick skin too.
>
>
> Ill give some background next and then a proposal.
>
> Background:
> I recently changed over to using authentication in the 1.5 snapshot
> because I need to have a session via the REST api so that I can set the
> session storage options in an initial query for a subsequent CTAS query.
> Previously all rest calls seemed to be completely independent.
>
> Since the change I have started seeing ‘too many files open’ errors in my
> drillbit.log and the drillbit java process becomes effectively hung waiting
> for open file descriptor slots. When running the top command the machine
> is running at max load due to the drillbit process and the drillbit becomes
> effectively unresponsive, even the simple pages in the web console don’t
> respond. Investigating further it seems that there might be a file kept
> open per session by the drillbit process for the life of the session. I
> used the lsof unix command on the drillbit process and found a lot of unix
> pipes. Looking at the code it looks like these pipes could be for the
> communication between the web process and the rpc server, with one being
> allocated per session. I haven’t validated this, its just a guess after
> scanning the code. I had 1.4 running without this requirement and without
> ever seeing the error. It seems without authentication the number of open
> files is a non-issue for me, possibly due to sessions.
>
> I'm wondering if my guess about what is causing the ‘too many open files’
> error is plausible? Does anybody with a deeper understanding of the
> architecture have any comments on this?
>
> Proposal:
> Assuming sessions are the issue, I am making some changes to my rest
> client so that sessions are more effectively used and I can up the ulimit
> for the drillbit process for the linux user in hopes of mitigating this. I
> am effectively creating a rest client based session pool that resets
> session variables to defaults when the session gets reused. However, it
> seems hacky.
>
> Below is an idea for getting per request based settings which seems less
> hacky in the long term.
>
> Can I add a new array member to the query.json REST method in a backwards
> compatible way to set session level parameters in a single request?
> Currently a rest request via the api has a body like so:
> { “queryType”: “SQL”, “query” : “<drill query>”}
>
> id like to do the following
>
> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”:
> [“option_1_name”:”option_1_value”, “option_2_name”:”option_2_value”]}
>
> or even
>
> { “queryType”: “SQL”, “query” : “<drill query>”, “sessionSettings”: [“SET
> `option_name` = value”, “SET `option_name1` = value1”,“SET `option_name2` =
> value2”, “SET `option_name3` = value3”]}
>
> As far as I can tell drill is essentially stateless between queries right
> now except for session level system parameters and authentication. There
> aren’t any in memory temp tables or cursors or variables like PL/SQL or
> PSQL or other SQLs that would make it stateful.
>
> Given the stateless assumption, being able to set session level params on
> a per request basis would cover all of the cases that I might need. It
> looks relatively straight forward to add something to QueryWrapper to
> accept an optional query session settings section of the json packet and
> execute those ’SET' commands before the final query. This will work for
> me, as I can run without authentication in an ’secure' backend environment
> which will remove sessions and hence file descriptors, assuming my
> assumptions about file descriptors and sessions are correct.
>
>
> My java is rusty (circa 2003) but some casual googling implies that if
> this were added as a 3rd @FormParam to submitQuery in QueryResources it
> would be magically be null if it werent present and could easily be
> ignored. If its present then an alternative constructor of QueryWrapper
> could be called with the extra param and it would be easy to alter its run
> method to execute the SET commands. There would need to be some error
> handling of course if the SET commands were illegal or failed to run for
> some reason.
>
> If this seems reasonable, how do I go about contributing? I looked
> through the links in the docs to apache foundation incubator projects but
> the links to drill were broken :( http://drill.apache.org/team.html <
> http://drill.apache.org/team.html> I read this
> http://drill.apache.org/docs/apache-drill-contribution-guidelines/ <
> http://drill.apache.org/docs/apache-drill-contribution-guidelines/> and
> i have subscribed to the dev mailing list (obvious since you are getting
> this). It said to post here before creating a JIRA. Am I missing
> anything in my assumptions? Comments? Should I just submit a JIRA and a
> patch or submit a JIRA and a comment or wait for comments before coding
> stuff up as an example?
>
> Thanks for taking the time to read and respond.
>
> Josh