You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Adam Kocoloski <ko...@apache.org> on 2016/10/07 01:57:44 UTC
Checking for cluster peers using SRV records on startup
Hi all,
Lately I’ve been thinking about how to ease the onramp for users to get a clustered CouchDB setup running. I think the Kubernetes work shows a lot of promise. One of the aspects of that work is the service discovery element; each node in a cluster should be able to automatically find its peers and connect to them. Kubernetes accomplishes this using SRV records; a DNS lookup for a given named service will return the FQDNs of all the live members of the “Pet Set”.
The SRV approach is enough of a standard[1] that I wonder if we ought to code for it directly in mem3. It’d eliminate the need for a “sidecar” container in Kubernetes deployments and I can imagine that it will prove more generally useful. The idea would be for mem3 to check if the CouchDB node is running in distributed node, and if it is, fire off a DNS lookup on the domain name, then attempt to connect with any other targets that are included in the record set in the DNS response.
What do you think? If no one objects I’ll file a JIRA and see what we come up with.
Adam
[1]: https://tools.ietf.org/html/rfc2782
Re: Checking for cluster peers using SRV records on startup
Posted by Adam Kocoloski <ad...@kocolosk.net>.
> On Nov 14, 2016, at 4:10 PM, Dave Cottlehuber <dc...@skunkwerks.at> wrote:
>
>> On Fri, 7 Oct 2016, at 02:57, Adam Kocoloski wrote:
>> Hi all,
>>
>> Lately I’ve been thinking about how to ease the onramp for users to get a
>> clustered CouchDB setup running. I think the Kubernetes work shows a lot
>> of promise. One of the aspects of that work is the service discovery
>> element; each node in a cluster should be able to automatically find its
>> peers and connect to them. Kubernetes accomplishes this using SRV
>> records; a DNS lookup for a given named service will return the FQDNs of
>> all the live members of the “Pet Set”.
>>
>> The SRV approach is enough of a standard[1] that I wonder if we ought to
>> code for it directly in mem3. It’d eliminate the need for a “sidecar”
>> container in Kubernetes deployments and I can imagine that it will prove
>> more generally useful. The idea would be for mem3 to check if the CouchDB
>> node is running in distributed node, and if it is, fire off a DNS lookup
>> on the domain name, then attempt to connect with any other targets that
>> are included in the record set in the DNS response.
>>
>> What do you think? If no one objects I’ll file a JIRA and see what we
>> come up with.
>
> belated +1 for _SRV records, +1000 for combining that with MDNS also for
> initial cluster setup. There are a number of erlang libraries already to
> help with this - below.
>
> the key issue with _SRV records is that, just like A records and CNAMEs
> is that you need to put them manually into DNS already. I am not sure
> this makes discovery easier...
>
> Something like mdns provides a multicast DNS lookup service, and nails
> this problem squarely.
>
> It is the bonjour/rendezvous/zeroconf tech used heavily by Apple
> devices, HP for printers etc. super nice. I am using this atm for
> service discovery albeit non-erlang stuff. It didnt quite get out of the
> RFC stage but its the defacto standard at least for consumer gear.
> http://www.zeroconf.org/
>
> https://github.com/shortishly/mdns or
> https://github.com/Licenser/erlang-mdns-server | client
>
> Also worth a look is https://github.com/shortishly/haystack with more
> docker sauce
>
> Another alternative is zbeacon, http://czmq.zeromq.org/manual:zbeacon ,
> also with erlang implementation https://github.com/zeromq/ezmq which
> I've only given a passing glance to.
>
> A+
> Dave
Hi Dave,
That makes sense. You can see I'm coming at this from the Kubernetes point of view, and in that world Kube itself takes care of populating the SRV records to automate the service discovery. I agree that if you have to populate the records manually you're not really gaining much. Cheers,
Adam
Re: Checking for cluster peers using SRV records on startup
Posted by Dave Cottlehuber <dc...@skunkwerks.at>.
On Fri, 7 Oct 2016, at 02:57, Adam Kocoloski wrote:
> Hi all,
>
> Lately I’ve been thinking about how to ease the onramp for users to get a
> clustered CouchDB setup running. I think the Kubernetes work shows a lot
> of promise. One of the aspects of that work is the service discovery
> element; each node in a cluster should be able to automatically find its
> peers and connect to them. Kubernetes accomplishes this using SRV
> records; a DNS lookup for a given named service will return the FQDNs of
> all the live members of the “Pet Set”.
>
> The SRV approach is enough of a standard[1] that I wonder if we ought to
> code for it directly in mem3. It’d eliminate the need for a “sidecar”
> container in Kubernetes deployments and I can imagine that it will prove
> more generally useful. The idea would be for mem3 to check if the CouchDB
> node is running in distributed node, and if it is, fire off a DNS lookup
> on the domain name, then attempt to connect with any other targets that
> are included in the record set in the DNS response.
>
> What do you think? If no one objects I’ll file a JIRA and see what we
> come up with.
belated +1 for _SRV records, +1000 for combining that with MDNS also for
initial cluster setup. There are a number of erlang libraries already to
help with this - below.
the key issue with _SRV records is that, just like A records and CNAMEs
is that you need to put them manually into DNS already. I am not sure
this makes discovery easier...
Something like mdns provides a multicast DNS lookup service, and nails
this problem squarely.
It is the bonjour/rendezvous/zeroconf tech used heavily by Apple
devices, HP for printers etc. super nice. I am using this atm for
service discovery albeit non-erlang stuff. It didnt quite get out of the
RFC stage but its the defacto standard at least for consumer gear.
http://www.zeroconf.org/
https://github.com/shortishly/mdns or
https://github.com/Licenser/erlang-mdns-server | client
Also worth a look is https://github.com/shortishly/haystack with more
docker sauce
Another alternative is zbeacon, http://czmq.zeromq.org/manual:zbeacon ,
also with erlang implementation https://github.com/zeromq/ezmq which
I've only given a passing glance to.
A+
Dave
Re: Checking for cluster peers using SRV records on startup
Posted by Robert Samuel Newson <rn...@apache.org>.
Hi Joe,
Ignore /_setup entirely, you're already well past that entry level. couchdb 2.0 and bigcouch should be about the same in your terms. The main difference is the 'nodes' db was renamed to '_nodes' to be more consistent with couch naming conventions, but you can change this back, as Cloudant does, with the .ini file config system.
> On 27 Oct 2016, at 01:40, Joe Black <jo...@valuphone.com> wrote:
>
> This would be great, as I’m running into massive issues trying to get couchdb to cluster in an automated fashion, with bigcouch this was no problem because you could simply use an init container on kubernetes to craft the http call to the /nodes endpoint to add the current nodes hostname, although adapting the http call to couchdb ie 5986/_nodes works from the init container the same, it only works for the initial cluster, after creating the initial cluster trying to add a new node by adding a new node to the nodes db fails to contact the node at any time in the future, but does create the record. This makes adding a node to a cluster in an automated fashion seemingly impossible. :(
>
> I have tried also using the official _setup endpoints but those fail completely because the add_node call is made before the new couch node is started, when failing to contact the node the entire operation seems to be ignored rather than trying again in the near future. I’m considering just going back to bigcouch since it’s broken ops automation entirely for us :(
>
> --
> Joe Black
>
>
> From: Adam Kocoloski <ko...@apache.org>
> Reply: dev@couchdb.apache.org <de...@couchdb.apache.org>>
> Date: October 6, 2016 at 9:57:55 PM
> To: dev@couchdb.apache.org <de...@couchdb.apache.org>>
> Subject: Checking for cluster peers using SRV records on startup
>
> Hi all,
>
> Lately I’ve been thinking about how to ease the onramp for users to get a clustered CouchDB setup running. I think the Kubernetes work shows a lot of promise. One of the aspects of that work is the service discovery element; each node in a cluster should be able to automatically find its peers and connect to them. Kubernetes accomplishes this using SRV records; a DNS lookup for a given named service will return the FQDNs of all the live members of the “Pet Set”.
>
> The SRV approach is enough of a standard[1] that I wonder if we ought to code for it directly in mem3. It’d eliminate the need for a “sidecar” container in Kubernetes deployments and I can imagine that it will prove more generally useful. The idea would be for mem3 to check if the CouchDB node is running in distributed node, and if it is, fire off a DNS lookup on the domain name, then attempt to connect with any other targets that are included in the record set in the DNS response.
>
> What do you think? If no one objects I’ll file a JIRA and see what we come up with.
>
> Adam
>
> [1]: https://tools.ietf.org/html/rfc2782
Re: Checking for cluster peers using SRV records on startup
Posted by Joe Black <jo...@valuphone.com>.
This would be great, as I’m running into massive issues trying to get couchdb to cluster in an automated fashion, with bigcouch this was no problem because you could simply use an init container on kubernetes to craft the http call to the /nodes endpoint to add the current nodes hostname, although adapting the http call to couchdb ie 5986/_nodes works from the init container the same, it only works for the initial cluster, after creating the initial cluster trying to add a new node by adding a new node to the nodes db fails to contact the node at any time in the future, but does create the record. This makes adding a node to a cluster in an automated fashion seemingly impossible. :(
I have tried also using the official _setup endpoints but those fail completely because the add_node call is made before the new couch node is started, when failing to contact the node the entire operation seems to be ignored rather than trying again in the near future. I’m considering just going back to bigcouch since it’s broken ops automation entirely for us :(
--
Joe Black
From: Adam Kocoloski <ko...@apache.org>
Reply: dev@couchdb.apache.org <de...@couchdb.apache.org>>
Date: October 6, 2016 at 9:57:55 PM
To: dev@couchdb.apache.org <de...@couchdb.apache.org>>
Subject: Checking for cluster peers using SRV records on startup
Hi all,
Lately I’ve been thinking about how to ease the onramp for users to get a clustered CouchDB setup running. I think the Kubernetes work shows a lot of promise. One of the aspects of that work is the service discovery element; each node in a cluster should be able to automatically find its peers and connect to them. Kubernetes accomplishes this using SRV records; a DNS lookup for a given named service will return the FQDNs of all the live members of the “Pet Set”.
The SRV approach is enough of a standard[1] that I wonder if we ought to code for it directly in mem3. It’d eliminate the need for a “sidecar” container in Kubernetes deployments and I can imagine that it will prove more generally useful. The idea would be for mem3 to check if the CouchDB node is running in distributed node, and if it is, fire off a DNS lookup on the domain name, then attempt to connect with any other targets that are included in the record set in the DNS response.
What do you think? If no one objects I’ll file a JIRA and see what we come up with.
Adam
[1]: https://tools.ietf.org/html/rfc2782
Re: Checking for cluster peers using SRV records on startup
Posted by Robert Samuel Newson <rn...@apache.org>.
I'm a fan of the SRV approach to service discovery. Everything else requires new/fiddly/custom infrastructure to support it.
> On 7 Oct 2016, at 02:57, Adam Kocoloski <ko...@apache.org> wrote:
>
> Hi all,
>
> Lately I’ve been thinking about how to ease the onramp for users to get a clustered CouchDB setup running. I think the Kubernetes work shows a lot of promise. One of the aspects of that work is the service discovery element; each node in a cluster should be able to automatically find its peers and connect to them. Kubernetes accomplishes this using SRV records; a DNS lookup for a given named service will return the FQDNs of all the live members of the “Pet Set”.
>
> The SRV approach is enough of a standard[1] that I wonder if we ought to code for it directly in mem3. It’d eliminate the need for a “sidecar” container in Kubernetes deployments and I can imagine that it will prove more generally useful. The idea would be for mem3 to check if the CouchDB node is running in distributed node, and if it is, fire off a DNS lookup on the domain name, then attempt to connect with any other targets that are included in the record set in the DNS response.
>
> What do you think? If no one objects I’ll file a JIRA and see what we come up with.
>
> Adam
>
> [1]: https://tools.ietf.org/html/rfc2782
Re: Checking for cluster peers using SRV records on startup
Posted by Paul Davis <pa...@gmail.com>.
I'd be for adding SRV records as long as we make it extensible so that
others can add support for whatever services they want as well. As SRV
records can be a bit problematic with larger clusters.
On Fri, Oct 7, 2016 at 3:37 PM, Andy Wenk <an...@apache.org> wrote:
> I never heard of SRV and had a overview look at the RFC. I believe this would be of great value including it and make it easier for new users to get a cluster setup running.
>
> +1
>
> Cheers
>
> Andy
>
> --
> Andy Wenk
> RockIt!
>
> Hamburg / Germany
>
> GPG public key: https://pgp.mit.edu/pks/lookup?op=get&search=0x4F1D0C59BC90917D
>
>> On 7 Oct 2016, at 19:08, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> This sounds useful, +1
>>
>> Best
>> Jan
>> --
>>
>>> On 07 Oct 2016, at 03:57, Adam Kocoloski <ko...@apache.org> wrote:
>>>
>>> Hi all,
>>>
>>> Lately I’ve been thinking about how to ease the onramp for users to get a clustered CouchDB setup running. I think the Kubernetes work shows a lot of promise. One of the aspects of that work is the service discovery element; each node in a cluster should be able to automatically find its peers and connect to them. Kubernetes accomplishes this using SRV records; a DNS lookup for a given named service will return the FQDNs of all the live members of the “Pet Set”.
>>>
>>> The SRV approach is enough of a standard[1] that I wonder if we ought to code for it directly in mem3. It’d eliminate the need for a “sidecar” container in Kubernetes deployments and I can imagine that it will prove more generally useful. The idea would be for mem3 to check if the CouchDB node is running in distributed node, and if it is, fire off a DNS lookup on the domain name, then attempt to connect with any other targets that are included in the record set in the DNS response.
>>>
>>> What do you think? If no one objects I’ll file a JIRA and see what we come up with.
>>>
>>> Adam
>>>
>>> [1]: https://tools.ietf.org/html/rfc2782
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
>
Re: Checking for cluster peers using SRV records on startup
Posted by Andy Wenk <an...@apache.org>.
I never heard of SRV and had a overview look at the RFC. I believe this would be of great value including it and make it easier for new users to get a cluster setup running.
+1
Cheers
Andy
--
Andy Wenk
RockIt!
Hamburg / Germany
GPG public key: https://pgp.mit.edu/pks/lookup?op=get&search=0x4F1D0C59BC90917D
> On 7 Oct 2016, at 19:08, Jan Lehnardt <ja...@apache.org> wrote:
>
> This sounds useful, +1
>
> Best
> Jan
> --
>
>> On 07 Oct 2016, at 03:57, Adam Kocoloski <ko...@apache.org> wrote:
>>
>> Hi all,
>>
>> Lately I’ve been thinking about how to ease the onramp for users to get a clustered CouchDB setup running. I think the Kubernetes work shows a lot of promise. One of the aspects of that work is the service discovery element; each node in a cluster should be able to automatically find its peers and connect to them. Kubernetes accomplishes this using SRV records; a DNS lookup for a given named service will return the FQDNs of all the live members of the “Pet Set”.
>>
>> The SRV approach is enough of a standard[1] that I wonder if we ought to code for it directly in mem3. It’d eliminate the need for a “sidecar” container in Kubernetes deployments and I can imagine that it will prove more generally useful. The idea would be for mem3 to check if the CouchDB node is running in distributed node, and if it is, fire off a DNS lookup on the domain name, then attempt to connect with any other targets that are included in the record set in the DNS response.
>>
>> What do you think? If no one objects I’ll file a JIRA and see what we come up with.
>>
>> Adam
>>
>> [1]: https://tools.ietf.org/html/rfc2782
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
Re: Checking for cluster peers using SRV records on startup
Posted by Jan Lehnardt <ja...@apache.org>.
This sounds useful, +1
Best
Jan
--
> On 07 Oct 2016, at 03:57, Adam Kocoloski <ko...@apache.org> wrote:
>
> Hi all,
>
> Lately I’ve been thinking about how to ease the onramp for users to get a clustered CouchDB setup running. I think the Kubernetes work shows a lot of promise. One of the aspects of that work is the service discovery element; each node in a cluster should be able to automatically find its peers and connect to them. Kubernetes accomplishes this using SRV records; a DNS lookup for a given named service will return the FQDNs of all the live members of the “Pet Set”.
>
> The SRV approach is enough of a standard[1] that I wonder if we ought to code for it directly in mem3. It’d eliminate the need for a “sidecar” container in Kubernetes deployments and I can imagine that it will prove more generally useful. The idea would be for mem3 to check if the CouchDB node is running in distributed node, and if it is, fire off a DNS lookup on the domain name, then attempt to connect with any other targets that are included in the record set in the DNS response.
>
> What do you think? If no one objects I’ll file a JIRA and see what we come up with.
>
> Adam
>
> [1]: https://tools.ietf.org/html/rfc2782
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/