You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by John Omernik <jo...@omernik.com> on 2017/06/22 19:14:05 UTC
Drill Session ID between Nodes
When I log onto a drill node, and get Session Id, if I connect to another
drill node in the cluster will the session id be valid?
I am guessing not, but want to validate.
My conumdrum, I have my Drill cluster running in such a way that the
connections to the nodes are load balanced via DNS. However, if I get a DNS
IP while in session it appears to invalidate, and thus forces me to log
on...
Re: Drill Session ID between Nodes
Posted by Paul Rogers <pr...@mapr.com>.
I stand corrected on one point (Thanks, Sorabh!): the Drill web server does have a session timeout, configurable in boot options, that defaults to one hour.
- Paul
> On Jun 23, 2017, at 2:10 PM, Paul Rogers <pr...@mapr.com> wrote:
>
> Hi John,
>
> Your use case is interesting. I’m certainly not an expert in the network aspects of what you are trying to do, but I can take a short at the related Drill issues.
>
> Drill’s primary use case is connecting via the Drill client (typically via JDBC or ODBC.) The Drill client handles security. It also allows SQL sessions, and hence session options.
>
> Your use case is based on the REST API. At present, the REST API is best described as a “prototype.” REST supports username/password login, and sessions associated with the login (on a single Drillbit). Sessions never timeout (as far as I can tell.) More importantly, the REST API returns all query results in a single message, encoded as JSON. This is great for small queries, but does not scale well when returning millions of large rows. (Hint: we are looking for contributions to improve the REST API!)
>
> As Keys pointed out, the important question is this: does your app need session state other than security? If so, then you need to consider overall SQL session state, not just SSL connections. If your script does “ALTER SESSION” followed by a query, then the ALTER SESSION might be sent to node A, with the query going to node B. Node B does not know about the session on A, and so results will be different than what you expect. The same is true with temp tables.
>
> Said another way, you’d like your scripts to do round-robin per *request*, but Drill is designed to do round-robin *per session.* (The Drill client, when using ZK, does random selection of nodes that achieves roughly the same result.) In short, your use case is clear, but is not supported today in Drill.
>
> Putting this together:
>
> 1. Sessions must be sticky to a single Drillbit so that session state, temp tables and so on are persisted (on that one Drillbit.)
> 2. If a session on one Drillbit drops, the app must establish a new session on another Drillbit. That involves not just security tokens and cookies, but also resetting session options, rebuilding temp tables, etc.
> 3. Since the app has to handle session recreation when switching Drillbits, the security issue, while a nuisance, is a necessary result of switching sessions.
> 4. (As Keys points out,) changing sessions is a rare event (due to timeouts, node failures, etc.) so session recovery should be rare.
>
> The only way to make sessions “portable” is to create a shared, global session shared across Drillbits, which is what you are proposing. Doing so is non-trivial: it requires a global session registry (or a way of synchronizing session state). Such sharing is not supported in Drill’s distributed, shared-nothing architecture. Could we add it? Probably, but not in the short term. If we ever find the need for a “metastore” (or central work scheduler), then at that time Drill would have a mechanism to support session portability; but that is a ways off.
>
> For the short term, can you perhaps rethink the use case given that sessions are local? How will your app handle failover? Is the security issue as much of a problem when seen as part of session recreation? (I’m not an expert here; I’m asking how this might work: are there things, short of persistent sessions, we can do to help?)
>
> You mentioned Drill-on-YARN (DoY). DoY is an interesting question. On the surface, REST works the same on DoY as in “regular” Drill: the REST endpoint doesn’t care how the Drillbit was launched. Whatever works with regular Drill will work with DoY. Under DOY, J/ODBC clients work as usual: they maintain a session with one Drillbit, and use ZK to find a fall-back Drillbit if the first one fails (with the need for the client to re-establish the SQL session state by resending session options, etc.) Can we improve this? Yes, if we did the work described earlier.
>
> (BTW: I’m still looking for volunteers to help with code reviews so we can contribute DoY to Apache Drill…)
>
> We have not yet looked into the security setup for DoY. (We wanted to get the security fully working with Drill itself first.) You raise some good issues that we must wrestle with as we enhance DoY to use the security features that are becoming available in Drill itself.
>
> Thanks,
>
> - Paul
>
>
>> On Jun 23, 2017, at 9:50 AM, John Omernik <jo...@omernik.com> wrote:
>>
>> That makes sense, ya, I would love to hear about the challenges of this in
>> general from the Drill folks.
>>
>> Also, I wonder if Paul R at MapR has any thoughts in how something like
>> this would be handled in the Drill on Yarn Setup.
>>
>>
>> John
>>
>
Re: Drill Session ID between Nodes
Posted by Paul Rogers <pr...@mapr.com>.
Hi John,
Your use case is interesting. I’m certainly not an expert in the network aspects of what you are trying to do, but I can take a short at the related Drill issues.
Drill’s primary use case is connecting via the Drill client (typically via JDBC or ODBC.) The Drill client handles security. It also allows SQL sessions, and hence session options.
Your use case is based on the REST API. At present, the REST API is best described as a “prototype.” REST supports username/password login, and sessions associated with the login (on a single Drillbit). Sessions never timeout (as far as I can tell.) More importantly, the REST API returns all query results in a single message, encoded as JSON. This is great for small queries, but does not scale well when returning millions of large rows. (Hint: we are looking for contributions to improve the REST API!)
As Keys pointed out, the important question is this: does your app need session state other than security? If so, then you need to consider overall SQL session state, not just SSL connections. If your script does “ALTER SESSION” followed by a query, then the ALTER SESSION might be sent to node A, with the query going to node B. Node B does not know about the session on A, and so results will be different than what you expect. The same is true with temp tables.
Said another way, you’d like your scripts to do round-robin per *request*, but Drill is designed to do round-robin *per session.* (The Drill client, when using ZK, does random selection of nodes that achieves roughly the same result.) In short, your use case is clear, but is not supported today in Drill.
Putting this together:
1. Sessions must be sticky to a single Drillbit so that session state, temp tables and so on are persisted (on that one Drillbit.)
2. If a session on one Drillbit drops, the app must establish a new session on another Drillbit. That involves not just security tokens and cookies, but also resetting session options, rebuilding temp tables, etc.
3. Since the app has to handle session recreation when switching Drillbits, the security issue, while a nuisance, is a necessary result of switching sessions.
4. (As Keys points out,) changing sessions is a rare event (due to timeouts, node failures, etc.) so session recovery should be rare.
The only way to make sessions “portable” is to create a shared, global session shared across Drillbits, which is what you are proposing. Doing so is non-trivial: it requires a global session registry (or a way of synchronizing session state). Such sharing is not supported in Drill’s distributed, shared-nothing architecture. Could we add it? Probably, but not in the short term. If we ever find the need for a “metastore” (or central work scheduler), then at that time Drill would have a mechanism to support session portability; but that is a ways off.
For the short term, can you perhaps rethink the use case given that sessions are local? How will your app handle failover? Is the security issue as much of a problem when seen as part of session recreation? (I’m not an expert here; I’m asking how this might work: are there things, short of persistent sessions, we can do to help?)
You mentioned Drill-on-YARN (DoY). DoY is an interesting question. On the surface, REST works the same on DoY as in “regular” Drill: the REST endpoint doesn’t care how the Drillbit was launched. Whatever works with regular Drill will work with DoY. Under DOY, J/ODBC clients work as usual: they maintain a session with one Drillbit, and use ZK to find a fall-back Drillbit if the first one fails (with the need for the client to re-establish the SQL session state by resending session options, etc.) Can we improve this? Yes, if we did the work described earlier.
(BTW: I’m still looking for volunteers to help with code reviews so we can contribute DoY to Apache Drill…)
We have not yet looked into the security setup for DoY. (We wanted to get the security fully working with Drill itself first.) You raise some good issues that we must wrestle with as we enhance DoY to use the security features that are becoming available in Drill itself.
Thanks,
- Paul
> On Jun 23, 2017, at 9:50 AM, John Omernik <jo...@omernik.com> wrote:
>
> That makes sense, ya, I would love to hear about the challenges of this in
> general from the Drill folks.
>
> Also, I wonder if Paul R at MapR has any thoughts in how something like
> this would be handled in the Drill on Yarn Setup.
>
>
> John
>
Re: Drill Session ID between Nodes
Posted by John Omernik <jo...@omernik.com>.
That makes sense, ya, I would love to hear about the challenges of this in
general from the Drill folks.
Also, I wonder if Paul R at MapR has any thoughts in how something like
this would be handled in the Drill on Yarn Setup.
John
On Fri, Jun 23, 2017 at 10:58 AM, Keys Botzum <kb...@mapr.com> wrote:
> I think we are on the same page regarding SSL.
>
> Regarding (1) it's best that I defer to the drill experts but I will
> mention that sharing session state can greatly complicate scalability.
> Since switching drillbits should be a rare event, it is probably more
> scalable to send back to the client a token which represents the
> authenticated identity (encrypted and signed of course). Then should that
> show up at another drillbit, the user authentication state can be
> reestablished. Other state such as caches would be lost of course. I don't
> know enough about Drill internals of course - there may be other state
> issues beyond just authentication.
>
> Keys
> _______________________________
> Keys Botzum
> Distinguished Engineer, Field Engineering
> kbotzum@maprtech.com<ma...@maprtech.com>
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Jun 23, 2017, at 11:33 AM, John Omernik <john@omernik.com<mailto:john@
> omernik.com>> wrote:
>
> So a few things
>
> 1. The issue is that as is, SSL stuff works fine, but when the IP address
> that DNS returns for the hostname changes, the session is invalidated and I
> am forced to logon again... this is annoying and loses session context
> information. If I try to lay out my cluster differently, i.e. using the
> wildcard certs and the different marathon layout, I then have different
> issues. I can connect by IP, but then I lose the SSL Validity. That's
> where the context for SSL comes into play. My main issue is with the IP
> returned for a DNS request changing during the course of a session,
> invalidating it. I think what it comes down to for me is this statement:
>
> As a user I connect to a drill cluster
>
> A simple statement, but what that means is as a user or an admin, my
> users/code accessing the cluster shouldn't have to care which individual
> node they connect to, they are connecting to a cluster. This is over
> simplifying things, but session ids managed by the cluster via Zookeeper
> would solve this.
>
>
> 2. I am looking at doing the SSL handling overrides n my python code,
> requests has some handlers for SSL and I was looking to address this,
> however, there is bug in how it works because it drops my custom port
> value... I am working on this now with the python requests folks. (i.e.
> the custom handlers would work, but only if I was connecting port 443)
>
>
>
>
>
> On Fri, Jun 23, 2017 at 9:52 AM, Keys Botzum <kbotzum@mapr.com<mailto:
> kbotzum@mapr.com>> wrote:
>
> There is something here I'm not understanding. In the below the hostname
> is always the same so there should be no problem as long as all drillbits
> share a common signer.
>
> I'm also just not following how certificate authentication issues are even
> linked to the Drill session issues. Whether or not there is a Drill
> session, the SSL handshake rules still apply. Or there is something here I
> just don't understand - quite possibly of course. I'm just focused on the
> SSL issue as this I understand very well.
>
> Incidentally, regarding hostname verification, I'm not familiar with what
> controls you have but many libraries (including Java) give you the ability
> to write your own SSL verifier which is called only when the default
> hostname verification fails. In that code you can implement different
> rules. Perhaps you can find a rule that meets your needs (such as a common
> signer for all Drillbits). Remember that certificate hostname validation is
> just a convention. There is nothing about SSL that makes this necessary.
> Here's the Java version: https://docs.oracle.com/
> javase/7/docs/api/javax/net/ssl/HostnameVerifier.html. In case you are
> curious, this is how MapR's maprlogin works with HTTPS even though we use
> IP address by default.
>
> Keys
> _______________________________
> Keys Botzum
> Distinguished Engineer, Field Engineering
> kbotzum@maprtech.com<ma...@maprtech.com><mailto:kbo
> tzum@maprtech.com>
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Jun 23, 2017, at 10:22 AM, John Omernik <john@omernik.com<mailto:john@
> omernik.com>> wrote:
>
> The wild card certificate isn't a problem on it's own, it's using it in a
> manner that allows me to maintain all of the various features I want. Let
> me lay this out,
>
> In marathon I have a task, it runs a drill bit. Since that task is located
> at the node prod/drillprod (for my env it's role/instanceid) the domain
> name is setup to be
>
> drillprod-prod.marathon.mesos.
>
> I can run X number of instances of that task. I tell Marathon to make them
> "host unique" so no two drill bits end up on the same node. This gives me
> a few things
>
> 1. If choose there to be 3 drillbits running, they go and run, and I don't
> have to worry about them. If I have to reboot the node one of them is on,
> Marathon says "oh look I am only running 2, let's spin up another, and then
> I get my required 3 bits running automatically.
>
> 2. They use a common config directory located in MapR-FS this is really
> nice because I don't have to maintain separate configurations for each
> drill bit.
>
> 3. The name above, drillprod-prod.marathon.mesos, using nslookup returns
>
> Name: drillprod-prod.marathon.mesos
>
> Address: 192.168.0.105
>
> Name: drillprod-prod.marathon.mesos
>
> Address: 192.168.0.103
>
> Name: drillprod-prod.marathon.mesos
>
> Address: 192.168.0.104
>
>
> Which is desired. When I have a client connect, I can program in a single
> name (drillprod-prod.marathon.mesos) into my script and never have to
> worry
> about where the bits run on the cluster. It looks it up and works great.
> This has been my standard MO for scripts that do short lived things... I
> haven't had an issue until this new use case came up. (long running
> sessions for use in analytic notebooks is the use case BTW, just not super
> relevant to go into details on that here)
>
>
> Because of the DNS naming, my scripts get tossed around to different bits
> depending on how the DNS round robin provides the IP which is desired for
> various scripts. The issue comes into play when I make a session
> connection, and for some reason, (maybe after a cache time out or
> something) python's requests object makes the next request, but does a DNS
> lookup first causing the IP to change, and the session to invalidate. Not
> awesome when working in a notebook.
>
> The wildcard DNS "could" work, but there are some gotchas... I could create
> an application folder in marathon with the same name, prod/drillprod, and
> then in there I could create a task with the hostname for each host.
>
> However, this would then make me loose on the HAness of my setup. If I am
> trying to run 3 instances of bits, on nodes node104, node103, and node105,
> and I need to reboot node105, in my setup, node102 could get the new bit
> the dns name auto updates and I maintain HA with simplicity, however, with
> a wildcard cert, I would still need to manually spin up a new instance to
> maintain three instances.
>
> In addition, I would have to get a list of the three nodes running to pick
> one to connect to. Lots of complex orchestration to use wild card certs to
> maintain HA.
>
> The reverse proxy will work for me, I can program nginx to pin
> connections. Thus, I can have it base which backend it goes to based on
> the JSESSIONID, that should work, but I don't like it because it requires
> another component running in my network, not bad for me, I can easily run
> that on Zeta that won't be an issue at all, but as a whole, it's not ideal
> for Drill users.
>
> Thus I am back to the idea of Drill somehow maintaining a global state.
> This is also important for Drill on Yarn setups (unless there is some sort
> of application container proxy back to the bits). If you want to have
> security (SSL) with hostnames, the session maintenance must be addressed.
>
> So that's why I toss it out here... this is a desirable feature I would
> imagine, even if people are not asking for it now, it may not because they
> don't need it, but in their testing of Drill, and how they using it now, it
> may not come up... when they have multiple people and services hitting
> drill end points pointing them individual nodes for SSL management etc,
> becomes a nightmare... thus, as a thought exercise, could be securely
> maintain valid session ideas in Zookeeper for nodes to check on? What would
> an ideal setup for something like that be?
>
>
>
>
> On Fri, Jun 23, 2017 at 7:07 AM, Keys Botzum <kbotzum@mapr.com<mailto:
> kbotzum@mapr.com>> wrote:
>
> Why is a wildcard certificate a problem? They are quite common. One just
> needs all of the Drillbits to share a common domain for the wildcard to be
> easy and thus avoid having to list individual hosts.
>
> Are you saying that you can't use hostnames and must use IPs?
>
> In case I'm not clear, here's an example of what I'm saying.
>
> this is good with wildcards: drill1.mydrill.corp.com<http:/
> /drill1.mydrill.corp.com><http:/
> /drill1.mydrill.corp.com<http://drill1.mydrill.corp.com>>,
> drill2.mydrill.corp.com<http://drill2.mydrill.corp.com><http:/
> /drill2.mydrill.corp.com<http://drill2.mydrill.corp.com>>,
> drill3.mydrill.corp.com<http://drill3.mydrill.corp.com><http:/
> /drill3.mydrill.corp.com<http://drill3.mydrill.corp.com>>,
> drill4.mydrill.corp.com<http://drill4.mydrill.corp.com><http:/
> /drill4.mydrill.corp.com<http://drill4.mydrill.corp.com>>,
> this is bad with wildcards: drill1, drill2, drill3, drill4
>
>
> Keys
> _______________________________
> Keys Botzum
> MapR Technologies
>
>
>
> On Jun 22, 2017, at 8:24 PM, John Omernik <john@omernik.com<mailto:john@
> omernik.com><mailto:john@
> omernik.com<http://omernik.com>>> wrote:
>
> Would there be interest in finding a way to globalize this? This is
> challenging for me and others that may run drill with multi Tennant
> orchestrators. In my particular setup, each node running drill gets added
> to an a record automatically giving me HA and distribution of Rest API
> queries. It also allows me to have a single certificate for my cluster
> rather than managing certificates on a individual basis. I set things up
> to connect via IP but then I had certificate mismatch warnings. My goal is
> to find a way to connect to the rest API , while maintaining a session to
> single node, with out sacrificing HA and balancing and with compromising
> ssl security. I know it's a tall order, but if there I ideas outside of a
> global state management I am all ears.
>
> Note some ideas I've also considered:
>
> 1. using a load balancer that would allow me to pin connections. Not
> ideal because it's another service to manage but it would work.
>
> 2. There may be a way to hack things with a wild card cert but it's seems
> complicated and fragile.
>
> On Jun 22, 2017 5:47 PM, "Sorabh Hamirwasia" <shamirwasia@mapr.com<mailto:
> shamirwasia@mapr.com><mailto:
> shamirwasia@mapr.com<ma...@mapr.com>>> wrote:
>
> Hi John,
> As Paul mentioned session ID's are not global. Each session is part of the
> BitToUserConnection instance created for a connection between Drillbit and
> client. Hence it's local to that Drillbit only and the lifetime of the
> session is tied to lifetime of the connection. You can find the code here<
> https://github.com/apache/drill/blob/master/exec/
> java-exec/src/main/java/org/apache/drill/exec/rpc/user/
> UserServer.java#L102>.
>
> Thanks,
> Sorabh
>
> ________________________________
> From: Paul Rogers <pr...@mapr.com>
> Sent: Thursday, June 22, 2017 2:19:50 PM
> To: user@drill.apache.org
> Subject: Re: Drill Session ID between Nodes
>
> Hi John,
>
> I do not believe that session IDs are global. Each Drillbit maintains its
> own concept of sessions. A global session would require some centralized
> registry of sessions, which Drill does not have.
>
> Would be great if someone can confirm…
>
> - Paul
>
> On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
>
> When I log onto a drill node, and get Session Id, if I connect to another
> drill node in the cluster will the session id be valid?
>
> I am guessing not, but want to validate.
>
> My conumdrum, I have my Drill cluster running in such a way that the
> connections to the nodes are load balanced via DNS. However, if I get a
> DNS
> IP while in session it appears to invalidate, and thus forces me to log
> on...
>
>
>
>
>
>
>
>
Re: Drill Session ID between Nodes
Posted by Keys Botzum <kb...@mapr.com>.
I think we are on the same page regarding SSL.
Regarding (1) it's best that I defer to the drill experts but I will mention that sharing session state can greatly complicate scalability. Since switching drillbits should be a rare event, it is probably more scalable to send back to the client a token which represents the authenticated identity (encrypted and signed of course). Then should that show up at another drillbit, the user authentication state can be reestablished. Other state such as caches would be lost of course. I don't know enough about Drill internals of course - there may be other state issues beyond just authentication.
Keys
_______________________________
Keys Botzum
Distinguished Engineer, Field Engineering
kbotzum@maprtech.com<ma...@maprtech.com>
443-718-0098
MapR Technologies
http://www.mapr.com
On Jun 23, 2017, at 11:33 AM, John Omernik <jo...@omernik.com>> wrote:
So a few things
1. The issue is that as is, SSL stuff works fine, but when the IP address
that DNS returns for the hostname changes, the session is invalidated and I
am forced to logon again... this is annoying and loses session context
information. If I try to lay out my cluster differently, i.e. using the
wildcard certs and the different marathon layout, I then have different
issues. I can connect by IP, but then I lose the SSL Validity. That's
where the context for SSL comes into play. My main issue is with the IP
returned for a DNS request changing during the course of a session,
invalidating it. I think what it comes down to for me is this statement:
As a user I connect to a drill cluster
A simple statement, but what that means is as a user or an admin, my
users/code accessing the cluster shouldn't have to care which individual
node they connect to, they are connecting to a cluster. This is over
simplifying things, but session ids managed by the cluster via Zookeeper
would solve this.
2. I am looking at doing the SSL handling overrides n my python code,
requests has some handlers for SSL and I was looking to address this,
however, there is bug in how it works because it drops my custom port
value... I am working on this now with the python requests folks. (i.e.
the custom handlers would work, but only if I was connecting port 443)
On Fri, Jun 23, 2017 at 9:52 AM, Keys Botzum <kb...@mapr.com>> wrote:
There is something here I'm not understanding. In the below the hostname
is always the same so there should be no problem as long as all drillbits
share a common signer.
I'm also just not following how certificate authentication issues are even
linked to the Drill session issues. Whether or not there is a Drill
session, the SSL handshake rules still apply. Or there is something here I
just don't understand - quite possibly of course. I'm just focused on the
SSL issue as this I understand very well.
Incidentally, regarding hostname verification, I'm not familiar with what
controls you have but many libraries (including Java) give you the ability
to write your own SSL verifier which is called only when the default
hostname verification fails. In that code you can implement different
rules. Perhaps you can find a rule that meets your needs (such as a common
signer for all Drillbits). Remember that certificate hostname validation is
just a convention. There is nothing about SSL that makes this necessary.
Here's the Java version: https://docs.oracle.com/
javase/7/docs/api/javax/net/ssl/HostnameVerifier.html. In case you are
curious, this is how MapR's maprlogin works with HTTPS even though we use
IP address by default.
Keys
_______________________________
Keys Botzum
Distinguished Engineer, Field Engineering
kbotzum@maprtech.com<ma...@maprtech.com>
443-718-0098
MapR Technologies
http://www.mapr.com
On Jun 23, 2017, at 10:22 AM, John Omernik <john@omernik.com<mailto:john@
omernik.com>> wrote:
The wild card certificate isn't a problem on it's own, it's using it in a
manner that allows me to maintain all of the various features I want. Let
me lay this out,
In marathon I have a task, it runs a drill bit. Since that task is located
at the node prod/drillprod (for my env it's role/instanceid) the domain
name is setup to be
drillprod-prod.marathon.mesos.
I can run X number of instances of that task. I tell Marathon to make them
"host unique" so no two drill bits end up on the same node. This gives me
a few things
1. If choose there to be 3 drillbits running, they go and run, and I don't
have to worry about them. If I have to reboot the node one of them is on,
Marathon says "oh look I am only running 2, let's spin up another, and then
I get my required 3 bits running automatically.
2. They use a common config directory located in MapR-FS this is really
nice because I don't have to maintain separate configurations for each
drill bit.
3. The name above, drillprod-prod.marathon.mesos, using nslookup returns
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.105
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.103
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.104
Which is desired. When I have a client connect, I can program in a single
name (drillprod-prod.marathon.mesos) into my script and never have to
worry
about where the bits run on the cluster. It looks it up and works great.
This has been my standard MO for scripts that do short lived things... I
haven't had an issue until this new use case came up. (long running
sessions for use in analytic notebooks is the use case BTW, just not super
relevant to go into details on that here)
Because of the DNS naming, my scripts get tossed around to different bits
depending on how the DNS round robin provides the IP which is desired for
various scripts. The issue comes into play when I make a session
connection, and for some reason, (maybe after a cache time out or
something) python's requests object makes the next request, but does a DNS
lookup first causing the IP to change, and the session to invalidate. Not
awesome when working in a notebook.
The wildcard DNS "could" work, but there are some gotchas... I could create
an application folder in marathon with the same name, prod/drillprod, and
then in there I could create a task with the hostname for each host.
However, this would then make me loose on the HAness of my setup. If I am
trying to run 3 instances of bits, on nodes node104, node103, and node105,
and I need to reboot node105, in my setup, node102 could get the new bit
the dns name auto updates and I maintain HA with simplicity, however, with
a wildcard cert, I would still need to manually spin up a new instance to
maintain three instances.
In addition, I would have to get a list of the three nodes running to pick
one to connect to. Lots of complex orchestration to use wild card certs to
maintain HA.
The reverse proxy will work for me, I can program nginx to pin
connections. Thus, I can have it base which backend it goes to based on
the JSESSIONID, that should work, but I don't like it because it requires
another component running in my network, not bad for me, I can easily run
that on Zeta that won't be an issue at all, but as a whole, it's not ideal
for Drill users.
Thus I am back to the idea of Drill somehow maintaining a global state.
This is also important for Drill on Yarn setups (unless there is some sort
of application container proxy back to the bits). If you want to have
security (SSL) with hostnames, the session maintenance must be addressed.
So that's why I toss it out here... this is a desirable feature I would
imagine, even if people are not asking for it now, it may not because they
don't need it, but in their testing of Drill, and how they using it now, it
may not come up... when they have multiple people and services hitting
drill end points pointing them individual nodes for SSL management etc,
becomes a nightmare... thus, as a thought exercise, could be securely
maintain valid session ideas in Zookeeper for nodes to check on? What would
an ideal setup for something like that be?
On Fri, Jun 23, 2017 at 7:07 AM, Keys Botzum <kbotzum@mapr.com<mailto:
kbotzum@mapr.com>> wrote:
Why is a wildcard certificate a problem? They are quite common. One just
needs all of the Drillbits to share a common domain for the wildcard to be
easy and thus avoid having to list individual hosts.
Are you saying that you can't use hostnames and must use IPs?
In case I'm not clear, here's an example of what I'm saying.
this is good with wildcards: drill1.mydrill.corp.com<http:/
/drill1.mydrill.corp.com><http:/
/drill1.mydrill.corp.com<http://drill1.mydrill.corp.com>>,
drill2.mydrill.corp.com<http://drill2.mydrill.corp.com><http:/
/drill2.mydrill.corp.com<http://drill2.mydrill.corp.com>>,
drill3.mydrill.corp.com<http://drill3.mydrill.corp.com><http:/
/drill3.mydrill.corp.com<http://drill3.mydrill.corp.com>>,
drill4.mydrill.corp.com<http://drill4.mydrill.corp.com><http:/
/drill4.mydrill.corp.com<http://drill4.mydrill.corp.com>>,
this is bad with wildcards: drill1, drill2, drill3, drill4
Keys
_______________________________
Keys Botzum
MapR Technologies
On Jun 22, 2017, at 8:24 PM, John Omernik <john@omernik.com<mailto:john@
omernik.com><mailto:john@
omernik.com<http://omernik.com>>> wrote:
Would there be interest in finding a way to globalize this? This is
challenging for me and others that may run drill with multi Tennant
orchestrators. In my particular setup, each node running drill gets added
to an a record automatically giving me HA and distribution of Rest API
queries. It also allows me to have a single certificate for my cluster
rather than managing certificates on a individual basis. I set things up
to connect via IP but then I had certificate mismatch warnings. My goal is
to find a way to connect to the rest API , while maintaining a session to
single node, with out sacrificing HA and balancing and with compromising
ssl security. I know it's a tall order, but if there I ideas outside of a
global state management I am all ears.
Note some ideas I've also considered:
1. using a load balancer that would allow me to pin connections. Not
ideal because it's another service to manage but it would work.
2. There may be a way to hack things with a wild card cert but it's seems
complicated and fragile.
On Jun 22, 2017 5:47 PM, "Sorabh Hamirwasia" <shamirwasia@mapr.com<mailto:
shamirwasia@mapr.com><mailto:
shamirwasia@mapr.com<ma...@mapr.com>>> wrote:
Hi John,
As Paul mentioned session ID's are not global. Each session is part of the
BitToUserConnection instance created for a connection between Drillbit and
client. Hence it's local to that Drillbit only and the lifetime of the
session is tied to lifetime of the connection. You can find the code here<
https://github.com/apache/drill/blob/master/exec/
java-exec/src/main/java/org/apache/drill/exec/rpc/user/
UserServer.java#L102>.
Thanks,
Sorabh
________________________________
From: Paul Rogers <pr...@mapr.com>
Sent: Thursday, June 22, 2017 2:19:50 PM
To: user@drill.apache.org
Subject: Re: Drill Session ID between Nodes
Hi John,
I do not believe that session IDs are global. Each Drillbit maintains its
own concept of sessions. A global session would require some centralized
registry of sessions, which Drill does not have.
Would be great if someone can confirm…
- Paul
On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
When I log onto a drill node, and get Session Id, if I connect to another
drill node in the cluster will the session id be valid?
I am guessing not, but want to validate.
My conumdrum, I have my Drill cluster running in such a way that the
connections to the nodes are load balanced via DNS. However, if I get a
DNS
IP while in session it appears to invalidate, and thus forces me to log
on...
Re: Drill Session ID between Nodes
Posted by John Omernik <jo...@omernik.com>.
So a few things
1. The issue is that as is, SSL stuff works fine, but when the IP address
that DNS returns for the hostname changes, the session is invalidated and I
am forced to logon again... this is annoying and loses session context
information. If I try to lay out my cluster differently, i.e. using the
wildcard certs and the different marathon layout, I then have different
issues. I can connect by IP, but then I lose the SSL Validity. That's
where the context for SSL comes into play. My main issue is with the IP
returned for a DNS request changing during the course of a session,
invalidating it. I think what it comes down to for me is this statement:
As a user I connect to a drill cluster
A simple statement, but what that means is as a user or an admin, my
users/code accessing the cluster shouldn't have to care which individual
node they connect to, they are connecting to a cluster. This is over
simplifying things, but session ids managed by the cluster via Zookeeper
would solve this.
2. I am looking at doing the SSL handling overrides n my python code,
requests has some handlers for SSL and I was looking to address this,
however, there is bug in how it works because it drops my custom port
value... I am working on this now with the python requests folks. (i.e.
the custom handlers would work, but only if I was connecting port 443)
On Fri, Jun 23, 2017 at 9:52 AM, Keys Botzum <kb...@mapr.com> wrote:
> There is something here I'm not understanding. In the below the hostname
> is always the same so there should be no problem as long as all drillbits
> share a common signer.
>
> I'm also just not following how certificate authentication issues are even
> linked to the Drill session issues. Whether or not there is a Drill
> session, the SSL handshake rules still apply. Or there is something here I
> just don't understand - quite possibly of course. I'm just focused on the
> SSL issue as this I understand very well.
>
> Incidentally, regarding hostname verification, I'm not familiar with what
> controls you have but many libraries (including Java) give you the ability
> to write your own SSL verifier which is called only when the default
> hostname verification fails. In that code you can implement different
> rules. Perhaps you can find a rule that meets your needs (such as a common
> signer for all Drillbits). Remember that certificate hostname validation is
> just a convention. There is nothing about SSL that makes this necessary.
> Here's the Java version: https://docs.oracle.com/
> javase/7/docs/api/javax/net/ssl/HostnameVerifier.html. In case you are
> curious, this is how MapR's maprlogin works with HTTPS even though we use
> IP address by default.
>
> Keys
> _______________________________
> Keys Botzum
> Distinguished Engineer, Field Engineering
> kbotzum@maprtech.com<ma...@maprtech.com>
> 443-718-0098
> MapR Technologies
> http://www.mapr.com
>
>
>
> On Jun 23, 2017, at 10:22 AM, John Omernik <john@omernik.com<mailto:john@
> omernik.com>> wrote:
>
> The wild card certificate isn't a problem on it's own, it's using it in a
> manner that allows me to maintain all of the various features I want. Let
> me lay this out,
>
> In marathon I have a task, it runs a drill bit. Since that task is located
> at the node prod/drillprod (for my env it's role/instanceid) the domain
> name is setup to be
>
> drillprod-prod.marathon.mesos.
>
> I can run X number of instances of that task. I tell Marathon to make them
> "host unique" so no two drill bits end up on the same node. This gives me
> a few things
>
> 1. If choose there to be 3 drillbits running, they go and run, and I don't
> have to worry about them. If I have to reboot the node one of them is on,
> Marathon says "oh look I am only running 2, let's spin up another, and then
> I get my required 3 bits running automatically.
>
> 2. They use a common config directory located in MapR-FS this is really
> nice because I don't have to maintain separate configurations for each
> drill bit.
>
> 3. The name above, drillprod-prod.marathon.mesos, using nslookup returns
>
> Name: drillprod-prod.marathon.mesos
>
> Address: 192.168.0.105
>
> Name: drillprod-prod.marathon.mesos
>
> Address: 192.168.0.103
>
> Name: drillprod-prod.marathon.mesos
>
> Address: 192.168.0.104
>
>
> Which is desired. When I have a client connect, I can program in a single
> name (drillprod-prod.marathon.mesos) into my script and never have to
> worry
> about where the bits run on the cluster. It looks it up and works great.
> This has been my standard MO for scripts that do short lived things... I
> haven't had an issue until this new use case came up. (long running
> sessions for use in analytic notebooks is the use case BTW, just not super
> relevant to go into details on that here)
>
>
> Because of the DNS naming, my scripts get tossed around to different bits
> depending on how the DNS round robin provides the IP which is desired for
> various scripts. The issue comes into play when I make a session
> connection, and for some reason, (maybe after a cache time out or
> something) python's requests object makes the next request, but does a DNS
> lookup first causing the IP to change, and the session to invalidate. Not
> awesome when working in a notebook.
>
> The wildcard DNS "could" work, but there are some gotchas... I could create
> an application folder in marathon with the same name, prod/drillprod, and
> then in there I could create a task with the hostname for each host.
>
> However, this would then make me loose on the HAness of my setup. If I am
> trying to run 3 instances of bits, on nodes node104, node103, and node105,
> and I need to reboot node105, in my setup, node102 could get the new bit
> the dns name auto updates and I maintain HA with simplicity, however, with
> a wildcard cert, I would still need to manually spin up a new instance to
> maintain three instances.
>
> In addition, I would have to get a list of the three nodes running to pick
> one to connect to. Lots of complex orchestration to use wild card certs to
> maintain HA.
>
> The reverse proxy will work for me, I can program nginx to pin
> connections. Thus, I can have it base which backend it goes to based on
> the JSESSIONID, that should work, but I don't like it because it requires
> another component running in my network, not bad for me, I can easily run
> that on Zeta that won't be an issue at all, but as a whole, it's not ideal
> for Drill users.
>
> Thus I am back to the idea of Drill somehow maintaining a global state.
> This is also important for Drill on Yarn setups (unless there is some sort
> of application container proxy back to the bits). If you want to have
> security (SSL) with hostnames, the session maintenance must be addressed.
>
> So that's why I toss it out here... this is a desirable feature I would
> imagine, even if people are not asking for it now, it may not because they
> don't need it, but in their testing of Drill, and how they using it now, it
> may not come up... when they have multiple people and services hitting
> drill end points pointing them individual nodes for SSL management etc,
> becomes a nightmare... thus, as a thought exercise, could be securely
> maintain valid session ideas in Zookeeper for nodes to check on? What would
> an ideal setup for something like that be?
>
>
>
>
> On Fri, Jun 23, 2017 at 7:07 AM, Keys Botzum <kbotzum@mapr.com<mailto:
> kbotzum@mapr.com>> wrote:
>
> Why is a wildcard certificate a problem? They are quite common. One just
> needs all of the Drillbits to share a common domain for the wildcard to be
> easy and thus avoid having to list individual hosts.
>
> Are you saying that you can't use hostnames and must use IPs?
>
> In case I'm not clear, here's an example of what I'm saying.
>
> this is good with wildcards: drill1.mydrill.corp.com<http:/
> /drill1.mydrill.corp.com><http:/
> /drill1.mydrill.corp.com<http://drill1.mydrill.corp.com>>,
> drill2.mydrill.corp.com<http://drill2.mydrill.corp.com><http:/
> /drill2.mydrill.corp.com<http://drill2.mydrill.corp.com>>,
> drill3.mydrill.corp.com<http://drill3.mydrill.corp.com><http:/
> /drill3.mydrill.corp.com<http://drill3.mydrill.corp.com>>,
> drill4.mydrill.corp.com<http://drill4.mydrill.corp.com><http:/
> /drill4.mydrill.corp.com<http://drill4.mydrill.corp.com>>,
> this is bad with wildcards: drill1, drill2, drill3, drill4
>
>
> Keys
> _______________________________
> Keys Botzum
> MapR Technologies
>
>
>
> On Jun 22, 2017, at 8:24 PM, John Omernik <john@omernik.com<mailto:john@
> omernik.com><mailto:john@
> omernik.com<http://omernik.com>>> wrote:
>
> Would there be interest in finding a way to globalize this? This is
> challenging for me and others that may run drill with multi Tennant
> orchestrators. In my particular setup, each node running drill gets added
> to an a record automatically giving me HA and distribution of Rest API
> queries. It also allows me to have a single certificate for my cluster
> rather than managing certificates on a individual basis. I set things up
> to connect via IP but then I had certificate mismatch warnings. My goal is
> to find a way to connect to the rest API , while maintaining a session to
> single node, with out sacrificing HA and balancing and with compromising
> ssl security. I know it's a tall order, but if there I ideas outside of a
> global state management I am all ears.
>
> Note some ideas I've also considered:
>
> 1. using a load balancer that would allow me to pin connections. Not
> ideal because it's another service to manage but it would work.
>
> 2. There may be a way to hack things with a wild card cert but it's seems
> complicated and fragile.
>
> On Jun 22, 2017 5:47 PM, "Sorabh Hamirwasia" <shamirwasia@mapr.com<mailto:
> shamirwasia@mapr.com><mailto:
> shamirwasia@mapr.com<ma...@mapr.com>>> wrote:
>
> Hi John,
> As Paul mentioned session ID's are not global. Each session is part of the
> BitToUserConnection instance created for a connection between Drillbit and
> client. Hence it's local to that Drillbit only and the lifetime of the
> session is tied to lifetime of the connection. You can find the code here<
> https://github.com/apache/drill/blob/master/exec/
> java-exec/src/main/java/org/apache/drill/exec/rpc/user/
> UserServer.java#L102>.
>
> Thanks,
> Sorabh
>
> ________________________________
> From: Paul Rogers <pr...@mapr.com>
> Sent: Thursday, June 22, 2017 2:19:50 PM
> To: user@drill.apache.org
> Subject: Re: Drill Session ID between Nodes
>
> Hi John,
>
> I do not believe that session IDs are global. Each Drillbit maintains its
> own concept of sessions. A global session would require some centralized
> registry of sessions, which Drill does not have.
>
> Would be great if someone can confirm…
>
> - Paul
>
> On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
>
> When I log onto a drill node, and get Session Id, if I connect to another
> drill node in the cluster will the session id be valid?
>
> I am guessing not, but want to validate.
>
> My conumdrum, I have my Drill cluster running in such a way that the
> connections to the nodes are load balanced via DNS. However, if I get a
> DNS
> IP while in session it appears to invalidate, and thus forces me to log
> on...
>
>
>
>
>
>
Re: Drill Session ID between Nodes
Posted by Keys Botzum <kb...@mapr.com>.
There is something here I'm not understanding. In the below the hostname is always the same so there should be no problem as long as all drillbits share a common signer.
I'm also just not following how certificate authentication issues are even linked to the Drill session issues. Whether or not there is a Drill session, the SSL handshake rules still apply. Or there is something here I just don't understand - quite possibly of course. I'm just focused on the SSL issue as this I understand very well.
Incidentally, regarding hostname verification, I'm not familiar with what controls you have but many libraries (including Java) give you the ability to write your own SSL verifier which is called only when the default hostname verification fails. In that code you can implement different rules. Perhaps you can find a rule that meets your needs (such as a common signer for all Drillbits). Remember that certificate hostname validation is just a convention. There is nothing about SSL that makes this necessary. Here's the Java version: https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/HostnameVerifier.html. In case you are curious, this is how MapR's maprlogin works with HTTPS even though we use IP address by default.
Keys
_______________________________
Keys Botzum
Distinguished Engineer, Field Engineering
kbotzum@maprtech.com<ma...@maprtech.com>
443-718-0098
MapR Technologies
http://www.mapr.com
On Jun 23, 2017, at 10:22 AM, John Omernik <jo...@omernik.com>> wrote:
The wild card certificate isn't a problem on it's own, it's using it in a
manner that allows me to maintain all of the various features I want. Let
me lay this out,
In marathon I have a task, it runs a drill bit. Since that task is located
at the node prod/drillprod (for my env it's role/instanceid) the domain
name is setup to be
drillprod-prod.marathon.mesos.
I can run X number of instances of that task. I tell Marathon to make them
"host unique" so no two drill bits end up on the same node. This gives me
a few things
1. If choose there to be 3 drillbits running, they go and run, and I don't
have to worry about them. If I have to reboot the node one of them is on,
Marathon says "oh look I am only running 2, let's spin up another, and then
I get my required 3 bits running automatically.
2. They use a common config directory located in MapR-FS this is really
nice because I don't have to maintain separate configurations for each
drill bit.
3. The name above, drillprod-prod.marathon.mesos, using nslookup returns
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.105
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.103
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.104
Which is desired. When I have a client connect, I can program in a single
name (drillprod-prod.marathon.mesos) into my script and never have to worry
about where the bits run on the cluster. It looks it up and works great.
This has been my standard MO for scripts that do short lived things... I
haven't had an issue until this new use case came up. (long running
sessions for use in analytic notebooks is the use case BTW, just not super
relevant to go into details on that here)
Because of the DNS naming, my scripts get tossed around to different bits
depending on how the DNS round robin provides the IP which is desired for
various scripts. The issue comes into play when I make a session
connection, and for some reason, (maybe after a cache time out or
something) python's requests object makes the next request, but does a DNS
lookup first causing the IP to change, and the session to invalidate. Not
awesome when working in a notebook.
The wildcard DNS "could" work, but there are some gotchas... I could create
an application folder in marathon with the same name, prod/drillprod, and
then in there I could create a task with the hostname for each host.
However, this would then make me loose on the HAness of my setup. If I am
trying to run 3 instances of bits, on nodes node104, node103, and node105,
and I need to reboot node105, in my setup, node102 could get the new bit
the dns name auto updates and I maintain HA with simplicity, however, with
a wildcard cert, I would still need to manually spin up a new instance to
maintain three instances.
In addition, I would have to get a list of the three nodes running to pick
one to connect to. Lots of complex orchestration to use wild card certs to
maintain HA.
The reverse proxy will work for me, I can program nginx to pin
connections. Thus, I can have it base which backend it goes to based on
the JSESSIONID, that should work, but I don't like it because it requires
another component running in my network, not bad for me, I can easily run
that on Zeta that won't be an issue at all, but as a whole, it's not ideal
for Drill users.
Thus I am back to the idea of Drill somehow maintaining a global state.
This is also important for Drill on Yarn setups (unless there is some sort
of application container proxy back to the bits). If you want to have
security (SSL) with hostnames, the session maintenance must be addressed.
So that's why I toss it out here... this is a desirable feature I would
imagine, even if people are not asking for it now, it may not because they
don't need it, but in their testing of Drill, and how they using it now, it
may not come up... when they have multiple people and services hitting
drill end points pointing them individual nodes for SSL management etc,
becomes a nightmare... thus, as a thought exercise, could be securely
maintain valid session ideas in Zookeeper for nodes to check on? What would
an ideal setup for something like that be?
On Fri, Jun 23, 2017 at 7:07 AM, Keys Botzum <kb...@mapr.com>> wrote:
Why is a wildcard certificate a problem? They are quite common. One just
needs all of the Drillbits to share a common domain for the wildcard to be
easy and thus avoid having to list individual hosts.
Are you saying that you can't use hostnames and must use IPs?
In case I'm not clear, here's an example of what I'm saying.
this is good with wildcards: drill1.mydrill.corp.com<http://drill1.mydrill.corp.com><http:/
/drill1.mydrill.corp.com<http://drill1.mydrill.corp.com>>, drill2.mydrill.corp.com<http://drill2.mydrill.corp.com><http:/
/drill2.mydrill.corp.com<http://drill2.mydrill.corp.com>>, drill3.mydrill.corp.com<http://drill3.mydrill.corp.com><http:/
/drill3.mydrill.corp.com<http://drill3.mydrill.corp.com>>, drill4.mydrill.corp.com<http://drill4.mydrill.corp.com><http:/
/drill4.mydrill.corp.com<http://drill4.mydrill.corp.com>>,
this is bad with wildcards: drill1, drill2, drill3, drill4
Keys
_______________________________
Keys Botzum
MapR Technologies
On Jun 22, 2017, at 8:24 PM, John Omernik <jo...@omernik.com><mailto:john@
omernik.com<http://omernik.com>>> wrote:
Would there be interest in finding a way to globalize this? This is
challenging for me and others that may run drill with multi Tennant
orchestrators. In my particular setup, each node running drill gets added
to an a record automatically giving me HA and distribution of Rest API
queries. It also allows me to have a single certificate for my cluster
rather than managing certificates on a individual basis. I set things up
to connect via IP but then I had certificate mismatch warnings. My goal is
to find a way to connect to the rest API , while maintaining a session to
single node, with out sacrificing HA and balancing and with compromising
ssl security. I know it's a tall order, but if there I ideas outside of a
global state management I am all ears.
Note some ideas I've also considered:
1. using a load balancer that would allow me to pin connections. Not
ideal because it's another service to manage but it would work.
2. There may be a way to hack things with a wild card cert but it's seems
complicated and fragile.
On Jun 22, 2017 5:47 PM, "Sorabh Hamirwasia" <sh...@mapr.com><mailto:
shamirwasia@mapr.com<ma...@mapr.com>>> wrote:
Hi John,
As Paul mentioned session ID's are not global. Each session is part of the
BitToUserConnection instance created for a connection between Drillbit and
client. Hence it's local to that Drillbit only and the lifetime of the
session is tied to lifetime of the connection. You can find the code here<
https://github.com/apache/drill/blob/master/exec/
java-exec/src/main/java/org/apache/drill/exec/rpc/user/
UserServer.java#L102>.
Thanks,
Sorabh
________________________________
From: Paul Rogers <pr...@mapr.com>
Sent: Thursday, June 22, 2017 2:19:50 PM
To: user@drill.apache.org
Subject: Re: Drill Session ID between Nodes
Hi John,
I do not believe that session IDs are global. Each Drillbit maintains its
own concept of sessions. A global session would require some centralized
registry of sessions, which Drill does not have.
Would be great if someone can confirm…
- Paul
On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
When I log onto a drill node, and get Session Id, if I connect to another
drill node in the cluster will the session id be valid?
I am guessing not, but want to validate.
My conumdrum, I have my Drill cluster running in such a way that the
connections to the nodes are load balanced via DNS. However, if I get a
DNS
IP while in session it appears to invalidate, and thus forces me to log
on...
Re: Drill Session ID between Nodes
Posted by John Omernik <jo...@omernik.com>.
The wild card certificate isn't a problem on it's own, it's using it in a
manner that allows me to maintain all of the various features I want. Let
me lay this out,
In marathon I have a task, it runs a drill bit. Since that task is located
at the node prod/drillprod (for my env it's role/instanceid) the domain
name is setup to be
drillprod-prod.marathon.mesos.
I can run X number of instances of that task. I tell Marathon to make them
"host unique" so no two drill bits end up on the same node. This gives me
a few things
1. If choose there to be 3 drillbits running, they go and run, and I don't
have to worry about them. If I have to reboot the node one of them is on,
Marathon says "oh look I am only running 2, let's spin up another, and then
I get my required 3 bits running automatically.
2. They use a common config directory located in MapR-FS this is really
nice because I don't have to maintain separate configurations for each
drill bit.
3. The name above, drillprod-prod.marathon.mesos, using nslookup returns
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.105
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.103
Name: drillprod-prod.marathon.mesos
Address: 192.168.0.104
Which is desired. When I have a client connect, I can program in a single
name (drillprod-prod.marathon.mesos) into my script and never have to worry
about where the bits run on the cluster. It looks it up and works great.
This has been my standard MO for scripts that do short lived things... I
haven't had an issue until this new use case came up. (long running
sessions for use in analytic notebooks is the use case BTW, just not super
relevant to go into details on that here)
Because of the DNS naming, my scripts get tossed around to different bits
depending on how the DNS round robin provides the IP which is desired for
various scripts. The issue comes into play when I make a session
connection, and for some reason, (maybe after a cache time out or
something) python's requests object makes the next request, but does a DNS
lookup first causing the IP to change, and the session to invalidate. Not
awesome when working in a notebook.
The wildcard DNS "could" work, but there are some gotchas... I could create
an application folder in marathon with the same name, prod/drillprod, and
then in there I could create a task with the hostname for each host.
However, this would then make me loose on the HAness of my setup. If I am
trying to run 3 instances of bits, on nodes node104, node103, and node105,
and I need to reboot node105, in my setup, node102 could get the new bit
the dns name auto updates and I maintain HA with simplicity, however, with
a wildcard cert, I would still need to manually spin up a new instance to
maintain three instances.
In addition, I would have to get a list of the three nodes running to pick
one to connect to. Lots of complex orchestration to use wild card certs to
maintain HA.
The reverse proxy will work for me, I can program nginx to pin
connections. Thus, I can have it base which backend it goes to based on
the JSESSIONID, that should work, but I don't like it because it requires
another component running in my network, not bad for me, I can easily run
that on Zeta that won't be an issue at all, but as a whole, it's not ideal
for Drill users.
Thus I am back to the idea of Drill somehow maintaining a global state.
This is also important for Drill on Yarn setups (unless there is some sort
of application container proxy back to the bits). If you want to have
security (SSL) with hostnames, the session maintenance must be addressed.
So that's why I toss it out here... this is a desirable feature I would
imagine, even if people are not asking for it now, it may not because they
don't need it, but in their testing of Drill, and how they using it now, it
may not come up... when they have multiple people and services hitting
drill end points pointing them individual nodes for SSL management etc,
becomes a nightmare... thus, as a thought exercise, could be securely
maintain valid session ideas in Zookeeper for nodes to check on? What would
an ideal setup for something like that be?
On Fri, Jun 23, 2017 at 7:07 AM, Keys Botzum <kb...@mapr.com> wrote:
> Why is a wildcard certificate a problem? They are quite common. One just
> needs all of the Drillbits to share a common domain for the wildcard to be
> easy and thus avoid having to list individual hosts.
>
> Are you saying that you can't use hostnames and must use IPs?
>
> In case I'm not clear, here's an example of what I'm saying.
>
> this is good with wildcards: drill1.mydrill.corp.com<http:/
> /drill1.mydrill.corp.com>, drill2.mydrill.corp.com<http:/
> /drill2.mydrill.corp.com>, drill3.mydrill.corp.com<http:/
> /drill3.mydrill.corp.com>, drill4.mydrill.corp.com<http:/
> /drill4.mydrill.corp.com>,
> this is bad with wildcards: drill1, drill2, drill3, drill4
>
>
> Keys
> _______________________________
> Keys Botzum
> MapR Technologies
>
>
>
> On Jun 22, 2017, at 8:24 PM, John Omernik <john@omernik.com<mailto:john@
> omernik.com>> wrote:
>
> Would there be interest in finding a way to globalize this? This is
> challenging for me and others that may run drill with multi Tennant
> orchestrators. In my particular setup, each node running drill gets added
> to an a record automatically giving me HA and distribution of Rest API
> queries. It also allows me to have a single certificate for my cluster
> rather than managing certificates on a individual basis. I set things up
> to connect via IP but then I had certificate mismatch warnings. My goal is
> to find a way to connect to the rest API , while maintaining a session to
> single node, with out sacrificing HA and balancing and with compromising
> ssl security. I know it's a tall order, but if there I ideas outside of a
> global state management I am all ears.
>
> Note some ideas I've also considered:
>
> 1. using a load balancer that would allow me to pin connections. Not
> ideal because it's another service to manage but it would work.
>
> 2. There may be a way to hack things with a wild card cert but it's seems
> complicated and fragile.
>
> On Jun 22, 2017 5:47 PM, "Sorabh Hamirwasia" <shamirwasia@mapr.com<mailto:
> shamirwasia@mapr.com>> wrote:
>
> Hi John,
> As Paul mentioned session ID's are not global. Each session is part of the
> BitToUserConnection instance created for a connection between Drillbit and
> client. Hence it's local to that Drillbit only and the lifetime of the
> session is tied to lifetime of the connection. You can find the code here<
> https://github.com/apache/drill/blob/master/exec/
> java-exec/src/main/java/org/apache/drill/exec/rpc/user/
> UserServer.java#L102>.
>
> Thanks,
> Sorabh
>
> ________________________________
> From: Paul Rogers <pr...@mapr.com>
> Sent: Thursday, June 22, 2017 2:19:50 PM
> To: user@drill.apache.org
> Subject: Re: Drill Session ID between Nodes
>
> Hi John,
>
> I do not believe that session IDs are global. Each Drillbit maintains its
> own concept of sessions. A global session would require some centralized
> registry of sessions, which Drill does not have.
>
> Would be great if someone can confirm…
>
> - Paul
>
> On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
>
> When I log onto a drill node, and get Session Id, if I connect to another
> drill node in the cluster will the session id be valid?
>
> I am guessing not, but want to validate.
>
> My conumdrum, I have my Drill cluster running in such a way that the
> connections to the nodes are load balanced via DNS. However, if I get a
> DNS
> IP while in session it appears to invalidate, and thus forces me to log
> on...
>
>
>
>
Re: Drill Session ID between Nodes
Posted by Keys Botzum <kb...@mapr.com>.
Why is a wildcard certificate a problem? They are quite common. One just needs all of the Drillbits to share a common domain for the wildcard to be easy and thus avoid having to list individual hosts.
Are you saying that you can't use hostnames and must use IPs?
In case I'm not clear, here's an example of what I'm saying.
this is good with wildcards: drill1.mydrill.corp.com<http://drill1.mydrill.corp.com>, drill2.mydrill.corp.com<http://drill2.mydrill.corp.com>, drill3.mydrill.corp.com<http://drill3.mydrill.corp.com>, drill4.mydrill.corp.com<http://drill4.mydrill.corp.com>,
this is bad with wildcards: drill1, drill2, drill3, drill4
Keys
_______________________________
Keys Botzum
MapR Technologies
On Jun 22, 2017, at 8:24 PM, John Omernik <jo...@omernik.com>> wrote:
Would there be interest in finding a way to globalize this? This is
challenging for me and others that may run drill with multi Tennant
orchestrators. In my particular setup, each node running drill gets added
to an a record automatically giving me HA and distribution of Rest API
queries. It also allows me to have a single certificate for my cluster
rather than managing certificates on a individual basis. I set things up
to connect via IP but then I had certificate mismatch warnings. My goal is
to find a way to connect to the rest API , while maintaining a session to
single node, with out sacrificing HA and balancing and with compromising
ssl security. I know it's a tall order, but if there I ideas outside of a
global state management I am all ears.
Note some ideas I've also considered:
1. using a load balancer that would allow me to pin connections. Not
ideal because it's another service to manage but it would work.
2. There may be a way to hack things with a wild card cert but it's seems
complicated and fragile.
On Jun 22, 2017 5:47 PM, "Sorabh Hamirwasia" <sh...@mapr.com>> wrote:
Hi John,
As Paul mentioned session ID's are not global. Each session is part of the
BitToUserConnection instance created for a connection between Drillbit and
client. Hence it's local to that Drillbit only and the lifetime of the
session is tied to lifetime of the connection. You can find the code here<
https://github.com/apache/drill/blob/master/exec/
java-exec/src/main/java/org/apache/drill/exec/rpc/user/
UserServer.java#L102>.
Thanks,
Sorabh
________________________________
From: Paul Rogers <pr...@mapr.com>
Sent: Thursday, June 22, 2017 2:19:50 PM
To: user@drill.apache.org
Subject: Re: Drill Session ID between Nodes
Hi John,
I do not believe that session IDs are global. Each Drillbit maintains its
own concept of sessions. A global session would require some centralized
registry of sessions, which Drill does not have.
Would be great if someone can confirm…
- Paul
On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
When I log onto a drill node, and get Session Id, if I connect to another
drill node in the cluster will the session id be valid?
I am guessing not, but want to validate.
My conumdrum, I have my Drill cluster running in such a way that the
connections to the nodes are load balanced via DNS. However, if I get a
DNS
IP while in session it appears to invalidate, and thus forces me to log
on...
Re: Drill Session ID between Nodes
Posted by John Omernik <jo...@omernik.com>.
Would there be interest in finding a way to globalize this? This is
challenging for me and others that may run drill with multi Tennant
orchestrators. In my particular setup, each node running drill gets added
to an a record automatically giving me HA and distribution of Rest API
queries. It also allows me to have a single certificate for my cluster
rather than managing certificates on a individual basis. I set things up
to connect via IP but then I had certificate mismatch warnings. My goal is
to find a way to connect to the rest API , while maintaining a session to
single node, with out sacrificing HA and balancing and with compromising
ssl security. I know it's a tall order, but if there I ideas outside of a
global state management I am all ears.
Note some ideas I've also considered:
1. using a load balancer that would allow me to pin connections. Not
ideal because it's another service to manage but it would work.
2. There may be a way to hack things with a wild card cert but it's seems
complicated and fragile.
On Jun 22, 2017 5:47 PM, "Sorabh Hamirwasia" <sh...@mapr.com> wrote:
> Hi John,
> As Paul mentioned session ID's are not global. Each session is part of the
> BitToUserConnection instance created for a connection between Drillbit and
> client. Hence it's local to that Drillbit only and the lifetime of the
> session is tied to lifetime of the connection. You can find the code here<
> https://github.com/apache/drill/blob/master/exec/
> java-exec/src/main/java/org/apache/drill/exec/rpc/user/
> UserServer.java#L102>.
>
> Thanks,
> Sorabh
>
> ________________________________
> From: Paul Rogers <pr...@mapr.com>
> Sent: Thursday, June 22, 2017 2:19:50 PM
> To: user@drill.apache.org
> Subject: Re: Drill Session ID between Nodes
>
> Hi John,
>
> I do not believe that session IDs are global. Each Drillbit maintains its
> own concept of sessions. A global session would require some centralized
> registry of sessions, which Drill does not have.
>
> Would be great if someone can confirm…
>
> - Paul
>
> > On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
> >
> > When I log onto a drill node, and get Session Id, if I connect to another
> > drill node in the cluster will the session id be valid?
> >
> > I am guessing not, but want to validate.
> >
> > My conumdrum, I have my Drill cluster running in such a way that the
> > connections to the nodes are load balanced via DNS. However, if I get a
> DNS
> > IP while in session it appears to invalidate, and thus forces me to log
> > on...
>
>
Re: Drill Session ID between Nodes
Posted by Sorabh Hamirwasia <sh...@mapr.com>.
Hi John,
As Paul mentioned session ID's are not global. Each session is part of the BitToUserConnection instance created for a connection between Drillbit and client. Hence it's local to that Drillbit only and the lifetime of the session is tied to lifetime of the connection. You can find the code here<https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java#L102>.
Thanks,
Sorabh
________________________________
From: Paul Rogers <pr...@mapr.com>
Sent: Thursday, June 22, 2017 2:19:50 PM
To: user@drill.apache.org
Subject: Re: Drill Session ID between Nodes
Hi John,
I do not believe that session IDs are global. Each Drillbit maintains its own concept of sessions. A global session would require some centralized registry of sessions, which Drill does not have.
Would be great if someone can confirm…
- Paul
> On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
>
> When I log onto a drill node, and get Session Id, if I connect to another
> drill node in the cluster will the session id be valid?
>
> I am guessing not, but want to validate.
>
> My conumdrum, I have my Drill cluster running in such a way that the
> connections to the nodes are load balanced via DNS. However, if I get a DNS
> IP while in session it appears to invalidate, and thus forces me to log
> on...
Re: Drill Session ID between Nodes
Posted by Paul Rogers <pr...@mapr.com>.
Hi John,
I do not believe that session IDs are global. Each Drillbit maintains its own concept of sessions. A global session would require some centralized registry of sessions, which Drill does not have.
Would be great if someone can confirm…
- Paul
> On Jun 22, 2017, at 12:14 PM, John Omernik <jo...@omernik.com> wrote:
>
> When I log onto a drill node, and get Session Id, if I connect to another
> drill node in the cluster will the session id be valid?
>
> I am guessing not, but want to validate.
>
> My conumdrum, I have my Drill cluster running in such a way that the
> connections to the nodes are load balanced via DNS. However, if I get a DNS
> IP while in session it appears to invalidate, and thus forces me to log
> on...