You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by John Sirois <js...@apache.org> on 2016/04/18 21:03:35 UTC
[PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
As part of work on switching the Aurora scheduler from using a copy of the
Twitter zookeeper libs to the Apache Curator libs [1], I'd like to enable a
Curator feature (GUID protection [2]) that will make the scheduler more
robust to ZooKeeper outages but has the side-effect of breaking the
effectively public API formed by the ZooKeeper path names it uses to
perform leader election and service advertisement. From Aurora's
standpoint, the API consumer is the Aurora command-line client. It uses
service-discovery today to find the leading scheduler to communicate with.
I propose switching the client to use HTTP re-directs instead since the
scheduler already has a redirect service and since this would reduce the
scheduler API surface area and generally move further down the road of a
pure HTTP api.
The most obvious problem I see here is just the mechanics of a proper
deprecation cycle for all those clients Aurora is not directly aware of
that rely on its service advertisment API in ZooKeeper today.
Are there other problems folks see with this?
[1] https://issues.apache.org/jira/browse/AURORA-1468
[2] https://issues.apache.org/jira/browse/AURORA-1670
Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
Posted by John Sirois <js...@apache.org>.
On Wed, Apr 20, 2016 at 1:26 AM, Erb, Stephan <St...@blue-yonder.com>
wrote:
> So, scheduler_uri would point to a domain name with multiple A-records?
Yes.
>
> We could probably also extend the interface to support a list of scheduler
> uris. That would make an initial HA-setup simpler, as it would not require
> the DNS entries or a load balancer. People could just use a list of IPs.
>
I was not thinking a load balancer would be needed if the transition is
from direct zk access today, the redirect following should amount to the
same availability. Folks that use an LB already, well they're all set.
I was waiting for someone to bring a list of IPs up. We could do this, but
the implication is folks access the Aurora UI via a fixed ip or a name like
aurora1.mycompany.com, aurora2,mycompany.com, ...
I was having a hard time seeing how access to the UI via 1 fixed ip/name
(that could be down even though the rest are up), was fine for the UI, but
not for the client.
> ________________________________________
> From: John Sirois <js...@apache.org>
> Sent: Tuesday, April 19, 2016 19:37
> To: Joshua Cohen
> Cc: dev@aurora.apache.org
> Subject: Re: [PROPOSAL] Switch aurora client from service discovery to
> HTTP redirects.
>
> On Mon, Apr 18, 2016 at 4:37 PM, Joshua Cohen <jc...@apache.org> wrote:
>
> > And apparently this is not part of our fork at all, the client already
> > supports this today! The only potential change that would be required
> would
> > be ensuring the client properly follows redirects.
> >
>
> Aha - yes. Thank you!
>
> Redirect follows are indeed not enabled on the requests calls today
> explicitly, but they are enabled implicitly for all non-HEAD requests IIUC.
>
> So I think there is a small amount of work to maybe test redirection
> actually works and deprecate the service discovery codepath in before
> removing support.
> Since there is support for the idea in the codebase already and you've
> indicated no objection I'll proceed to draw up some RBs.
>
>
> > On Mon, Apr 18, 2016 at 5:35 PM, Joshua Cohen <jc...@apache.org> wrote:
> >
> >> Er, it's not `proxy_url`, it's `scheduler_uri` (which makes much more
> >> sense ;)).
> >>
> >> On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jc...@apache.org>
> wrote:
> >>
> >>> I'm not opposed to this, in fact we already do something similar
> >>> internally. We've forked clusters.py to allow configuring a
> `proxy_url` for
> >>> each cluster. If that's present, then the client will use it rather
> than
> >>> performing service discovery to communicate with the scheduler.
> >>>
> >>> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <js...@apache.org>
> wrote:
> >>>
> >>>> As part of work on switching the Aurora scheduler from using a copy of
> >>>> the
> >>>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to
> >>>> enable a
> >>>> Curator feature (GUID protection [2]) that will make the scheduler
> more
> >>>> robust to ZooKeeper outages but has the side-effect of breaking the
> >>>> effectively public API formed by the ZooKeeper path names it uses to
> >>>> perform leader election and service advertisement. From Aurora's
> >>>> standpoint, the API consumer is the Aurora command-line client. It
> uses
> >>>> service-discovery today to find the leading scheduler to communicate
> >>>> with.
> >>>> I propose switching the client to use HTTP re-directs instead since
> the
> >>>> scheduler already has a redirect service and since this would reduce
> the
> >>>> scheduler API surface area and generally move further down the road
> of a
> >>>> pure HTTP api.
> >>>>
> >>>> The most obvious problem I see here is just the mechanics of a proper
> >>>> deprecation cycle for all those clients Aurora is not directly aware
> of
> >>>> that rely on its service advertisment API in ZooKeeper today.
> >>>>
> >>>> Are there other problems folks see with this?
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/AURORA-1468
> >>>> [2] https://issues.apache.org/jira/browse/AURORA-1670
> >>>>
> >>>
> >>>
> >>
> >
>
Re: [PROPOSAL] Switch aurora client from service discovery to HTTP
redirects.
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
So, scheduler_uri would point to a domain name with multiple A-records?
We could probably also extend the interface to support a list of scheduler uris. That would make an initial HA-setup simpler, as it would not require the DNS entries or a load balancer. People could just use a list of IPs.
________________________________________
From: John Sirois <js...@apache.org>
Sent: Tuesday, April 19, 2016 19:37
To: Joshua Cohen
Cc: dev@aurora.apache.org
Subject: Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
On Mon, Apr 18, 2016 at 4:37 PM, Joshua Cohen <jc...@apache.org> wrote:
> And apparently this is not part of our fork at all, the client already
> supports this today! The only potential change that would be required would
> be ensuring the client properly follows redirects.
>
Aha - yes. Thank you!
Redirect follows are indeed not enabled on the requests calls today
explicitly, but they are enabled implicitly for all non-HEAD requests IIUC.
So I think there is a small amount of work to maybe test redirection
actually works and deprecate the service discovery codepath in before
removing support.
Since there is support for the idea in the codebase already and you've
indicated no objection I'll proceed to draw up some RBs.
> On Mon, Apr 18, 2016 at 5:35 PM, Joshua Cohen <jc...@apache.org> wrote:
>
>> Er, it's not `proxy_url`, it's `scheduler_uri` (which makes much more
>> sense ;)).
>>
>> On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jc...@apache.org> wrote:
>>
>>> I'm not opposed to this, in fact we already do something similar
>>> internally. We've forked clusters.py to allow configuring a `proxy_url` for
>>> each cluster. If that's present, then the client will use it rather than
>>> performing service discovery to communicate with the scheduler.
>>>
>>> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <js...@apache.org> wrote:
>>>
>>>> As part of work on switching the Aurora scheduler from using a copy of
>>>> the
>>>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to
>>>> enable a
>>>> Curator feature (GUID protection [2]) that will make the scheduler more
>>>> robust to ZooKeeper outages but has the side-effect of breaking the
>>>> effectively public API formed by the ZooKeeper path names it uses to
>>>> perform leader election and service advertisement. From Aurora's
>>>> standpoint, the API consumer is the Aurora command-line client. It uses
>>>> service-discovery today to find the leading scheduler to communicate
>>>> with.
>>>> I propose switching the client to use HTTP re-directs instead since the
>>>> scheduler already has a redirect service and since this would reduce the
>>>> scheduler API surface area and generally move further down the road of a
>>>> pure HTTP api.
>>>>
>>>> The most obvious problem I see here is just the mechanics of a proper
>>>> deprecation cycle for all those clients Aurora is not directly aware of
>>>> that rely on its service advertisment API in ZooKeeper today.
>>>>
>>>> Are there other problems folks see with this?
>>>>
>>>> [1] https://issues.apache.org/jira/browse/AURORA-1468
>>>> [2] https://issues.apache.org/jira/browse/AURORA-1670
>>>>
>>>
>>>
>>
>
Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
Posted by John Sirois <js...@apache.org>.
On Mon, Apr 18, 2016 at 4:37 PM, Joshua Cohen <jc...@apache.org> wrote:
> And apparently this is not part of our fork at all, the client already
> supports this today! The only potential change that would be required would
> be ensuring the client properly follows redirects.
>
Aha - yes. Thank you!
Redirect follows are indeed not enabled on the requests calls today
explicitly, but they are enabled implicitly for all non-HEAD requests IIUC.
So I think there is a small amount of work to maybe test redirection
actually works and deprecate the service discovery codepath in before
removing support.
Since there is support for the idea in the codebase already and you've
indicated no objection I'll proceed to draw up some RBs.
> On Mon, Apr 18, 2016 at 5:35 PM, Joshua Cohen <jc...@apache.org> wrote:
>
>> Er, it's not `proxy_url`, it's `scheduler_uri` (which makes much more
>> sense ;)).
>>
>> On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jc...@apache.org> wrote:
>>
>>> I'm not opposed to this, in fact we already do something similar
>>> internally. We've forked clusters.py to allow configuring a `proxy_url` for
>>> each cluster. If that's present, then the client will use it rather than
>>> performing service discovery to communicate with the scheduler.
>>>
>>> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <js...@apache.org> wrote:
>>>
>>>> As part of work on switching the Aurora scheduler from using a copy of
>>>> the
>>>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to
>>>> enable a
>>>> Curator feature (GUID protection [2]) that will make the scheduler more
>>>> robust to ZooKeeper outages but has the side-effect of breaking the
>>>> effectively public API formed by the ZooKeeper path names it uses to
>>>> perform leader election and service advertisement. From Aurora's
>>>> standpoint, the API consumer is the Aurora command-line client. It uses
>>>> service-discovery today to find the leading scheduler to communicate
>>>> with.
>>>> I propose switching the client to use HTTP re-directs instead since the
>>>> scheduler already has a redirect service and since this would reduce the
>>>> scheduler API surface area and generally move further down the road of a
>>>> pure HTTP api.
>>>>
>>>> The most obvious problem I see here is just the mechanics of a proper
>>>> deprecation cycle for all those clients Aurora is not directly aware of
>>>> that rely on its service advertisment API in ZooKeeper today.
>>>>
>>>> Are there other problems folks see with this?
>>>>
>>>> [1] https://issues.apache.org/jira/browse/AURORA-1468
>>>> [2] https://issues.apache.org/jira/browse/AURORA-1670
>>>>
>>>
>>>
>>
>
Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
Posted by Joshua Cohen <jc...@apache.org>.
And apparently this is not part of our fork at all, the client already
supports this today! The only potential change that would be required would
be ensuring the client properly follows redirects.
On Mon, Apr 18, 2016 at 5:35 PM, Joshua Cohen <jc...@apache.org> wrote:
> Er, it's not `proxy_url`, it's `scheduler_uri` (which makes much more
> sense ;)).
>
> On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jc...@apache.org> wrote:
>
>> I'm not opposed to this, in fact we already do something similar
>> internally. We've forked clusters.py to allow configuring a `proxy_url` for
>> each cluster. If that's present, then the client will use it rather than
>> performing service discovery to communicate with the scheduler.
>>
>> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <js...@apache.org> wrote:
>>
>>> As part of work on switching the Aurora scheduler from using a copy of
>>> the
>>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to
>>> enable a
>>> Curator feature (GUID protection [2]) that will make the scheduler more
>>> robust to ZooKeeper outages but has the side-effect of breaking the
>>> effectively public API formed by the ZooKeeper path names it uses to
>>> perform leader election and service advertisement. From Aurora's
>>> standpoint, the API consumer is the Aurora command-line client. It uses
>>> service-discovery today to find the leading scheduler to communicate
>>> with.
>>> I propose switching the client to use HTTP re-directs instead since the
>>> scheduler already has a redirect service and since this would reduce the
>>> scheduler API surface area and generally move further down the road of a
>>> pure HTTP api.
>>>
>>> The most obvious problem I see here is just the mechanics of a proper
>>> deprecation cycle for all those clients Aurora is not directly aware of
>>> that rely on its service advertisment API in ZooKeeper today.
>>>
>>> Are there other problems folks see with this?
>>>
>>> [1] https://issues.apache.org/jira/browse/AURORA-1468
>>> [2] https://issues.apache.org/jira/browse/AURORA-1670
>>>
>>
>>
>
Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
Posted by Joshua Cohen <jc...@apache.org>.
Er, it's not `proxy_url`, it's `scheduler_uri` (which makes much more
sense ;)).
On Mon, Apr 18, 2016 at 5:34 PM, Joshua Cohen <jc...@apache.org> wrote:
> I'm not opposed to this, in fact we already do something similar
> internally. We've forked clusters.py to allow configuring a `proxy_url` for
> each cluster. If that's present, then the client will use it rather than
> performing service discovery to communicate with the scheduler.
>
> On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <js...@apache.org> wrote:
>
>> As part of work on switching the Aurora scheduler from using a copy of the
>> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to enable
>> a
>> Curator feature (GUID protection [2]) that will make the scheduler more
>> robust to ZooKeeper outages but has the side-effect of breaking the
>> effectively public API formed by the ZooKeeper path names it uses to
>> perform leader election and service advertisement. From Aurora's
>> standpoint, the API consumer is the Aurora command-line client. It uses
>> service-discovery today to find the leading scheduler to communicate with.
>> I propose switching the client to use HTTP re-directs instead since the
>> scheduler already has a redirect service and since this would reduce the
>> scheduler API surface area and generally move further down the road of a
>> pure HTTP api.
>>
>> The most obvious problem I see here is just the mechanics of a proper
>> deprecation cycle for all those clients Aurora is not directly aware of
>> that rely on its service advertisment API in ZooKeeper today.
>>
>> Are there other problems folks see with this?
>>
>> [1] https://issues.apache.org/jira/browse/AURORA-1468
>> [2] https://issues.apache.org/jira/browse/AURORA-1670
>>
>
>
Re: [PROPOSAL] Switch aurora client from service discovery to HTTP redirects.
Posted by Joshua Cohen <jc...@apache.org>.
I'm not opposed to this, in fact we already do something similar
internally. We've forked clusters.py to allow configuring a `proxy_url` for
each cluster. If that's present, then the client will use it rather than
performing service discovery to communicate with the scheduler.
On Mon, Apr 18, 2016 at 2:03 PM, John Sirois <js...@apache.org> wrote:
> As part of work on switching the Aurora scheduler from using a copy of the
> Twitter zookeeper libs to the Apache Curator libs [1], I'd like to enable a
> Curator feature (GUID protection [2]) that will make the scheduler more
> robust to ZooKeeper outages but has the side-effect of breaking the
> effectively public API formed by the ZooKeeper path names it uses to
> perform leader election and service advertisement. From Aurora's
> standpoint, the API consumer is the Aurora command-line client. It uses
> service-discovery today to find the leading scheduler to communicate with.
> I propose switching the client to use HTTP re-directs instead since the
> scheduler already has a redirect service and since this would reduce the
> scheduler API surface area and generally move further down the road of a
> pure HTTP api.
>
> The most obvious problem I see here is just the mechanics of a proper
> deprecation cycle for all those clients Aurora is not directly aware of
> that rely on its service advertisment API in ZooKeeper today.
>
> Are there other problems folks see with this?
>
> [1] https://issues.apache.org/jira/browse/AURORA-1468
> [2] https://issues.apache.org/jira/browse/AURORA-1670
>