You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jay Kreps <ja...@gmail.com> on 2014/10/17 23:28:48 UTC
blog with some "out of the box" pains
This guy documented a few struggles getting going with Kafka. Not sure if
there is anything we can do to make it better?
http://ispyker.blogspot.com/2014/10/kafka-part-1.html
1. Would be great to figure out the apache/gradle thing.
2. The problem of having Kafka advertise localhost on AWS is really common.
I was thinking one possible solution for this would be to get all the
interfaces and prefer non-localhost interfaces if they exist.
-Jay
Re: blog with some "out of the box" pains
Posted by Joe Stein <jo...@stealth.ly>.
1) Kafka Gradle thing would be great to figure out. Samza, Aurora (maybe a
few other Apache projects) would benefit too. We need a better way to
bootstrap the gradle-wrapper.jar.I created
https://issues.apache.org/jira/browse/KAFKA-1714 to track that.
2) I have seen, more than a few times, a Kafka deployment setting the
advertised.host.name as the same value (except with the ".") as broker.id (
host.name often not even used). That becomes especially helpful when using
cloud formation, chef, puppet, ansible, etc. We could auto-advertise the
broker list best better created
https://issues.apache.org/jira/browse/KAFKA-1715 for that.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Sat, Oct 18, 2014 at 12:03 AM, Ewen Cheslack-Postava <me...@ewencp.org>
wrote:
> The first issue he runs into is one I also find frustrating -- with
> cloud providers pushing SSDs, you have to use a pretty large instance
> type to get a reasonable test setup. I'm not sure if he couldn't launch
> an older type like m1.large (I think some newer AWS accounts aren't able
> to) or if he just didn't see it as an option since they are hidden by
> default. Even the largest general purpose instance types are pretty
> wimpy wrt storage, only 80GB local instance storage.
>
> The hostname issues are a well known pain point and unfortunately there
> aren't any great solutions that aren't EC2-specific. Here's a quick run
> down:
>
> * None of the images for popular distros on EC2 will auto-set the
> hostname beyond what EC2 already sets up (which isn't publicly
> routable). The following details might explain why they can't. For
> example, a recent Ubuntu image gives:
>
> ubuntu@ip-172-30-2-76:~$ hostname
> ip-172-30-2-76
>
> ubuntu@ip-172-30-2-76:~$ cat /etc/hosts
> 127.0.0.1 localhost
>
> # The following lines are desirable for IPv6 capable hosts
> ::1 ip6-localhost ip6-loopback
> --- cut irrelevant pieces ---
>
> * Sometimes the hostname is set, but isn't useful. For example, in this
> Ubuntu image, the hostname is set to "ip-[ip-address-]", but that isn't
> routable, so generates really irritating behavior. Running on the server
> itself (which is running in a VPC, see below for more details):
>
> scala> InetAddress.getLocalHost
> java.net.UnknownHostException: ip-172-30-2-76: ip-172-30-2-76: Name or
> service not known
> at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
> at .<init>(<console>:9)
> at .<clinit>(<console>)
> at .<init>(<console>:11)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
> scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:704)
> at
>
> scala.tools.nsc.interpreter.IMain$Request$$anonfun$14.apply(IMain.scala:920)
> at
>
> scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
> at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.UnknownHostException: ip-172-30-2-76: Name or
> service not known
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at
> java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
> at
>
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
> at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
> ... 14 more
>
> * As described in a bunch of places, the only reliable way to get public
> DNS info is through EC2's own instance metadata API:
> https://forums.aws.amazon.com/thread.jspa?threadID=77788 For example:
>
> curl -s http://169.254.169.254/latest/meta-data/public-hostname
>
> might give something like:
>
> ec2-203-0-113-25.compute-1.amazonaws.com
>
> * But you may not even *have* a public DNS hostname. If you launch in a
> VPC, you'll only get one if you set the VPC to generate them (and I'm
> pretty sure the default is to not create them):
> http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html The
> output of the curl call above will just be empty.
>
> * AWS is pretty aggressively trying to move away from EC2-Classic (i.e.
> non-VPC instances), so most new instances will end up in VPCs unless you
> are working in a grandfathered account + AZ. If VPC without public DNS
> is the default, we'll have to carefully guide new users in generating a
> setup that works properly if we try to use hostnames.
>
> * Even if you try moving the IP addresses, you still have to deal with
> VPCs. You can't directly get your public IP address without accessing
> something outside the host since you're in a VPC. You need to use the
> instance metadata API to look it up, i.e.,
>
> curl -s http://169.254.169.254/latest/meta-data/public-ipv4
>
> * And yet another problem with IPs: unless you use an elastic IP, you're
> not guaranteed they'll be stable:
>
> Auto-assign Public IP
>
> Requests a public IP address from Amazon's public IP address pool,
> to make your instance reachable from the Internet. In most cases, the
> public IP address is associated with the instance until it’s stopped
> or
> terminated, after which it’s no longer available for you to use. If
> you
> require a persistent public IP address that you can associate and
> disassociate at will, use an Elastic IP address (EIP) instead. You can
> allocate your own EIP, and associate it to your instance after launch.
>
> I know Spark had some similar issues -- using their (very convenient!)
> ec2 script, you still ended up with some stuff in their web interface
> that linked to internal addresses such that the links wouldn't work. I'm
> not sure if they have figured out a decent work around. But as you can
> see from the above, it's unlikely you can use generic approaches to get
> the info we need -- it'll need to be platform specific, which probably
> means it's better to determine it outside the main Kafka code and
> provide it via advertised.host.name.
>
> -Ewen
>
> On Fri, Oct 17, 2014, at 05:11 PM, Gwen Shapira wrote:
> > Basically, the issue (or at least one of very many possible network
> > issues...) is that the server has "localhost" hardcoded as its
> > canonical name in /etc/hosts:
> >
> > [root@Billc-cent70x64 ~]# cat /etc/hosts
> > 127.0.0.1 localhost localhost.localdomain localhost4
> > localhost4.localdomain4 Billc-cent70x64
> > ::1 localhost localhost.localdomain localhost6
> > localhost6.localdomain6
> >
> > Unfortunately a very common default for RedHat and Centos machines.
> >
> > As the blog mentions, a good solution (other than instructing Kafka on
> > the right name to advertise) is to add the correct IP and hostname to
> > /etc/hosts. We may want to add this option to the FAQ.
> >
> > Gwen
> >
> >
> >
> >
> > On Fri, Oct 17, 2014 at 7:56 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> > > It looks like we are using canonical hostname:
> > >
> > > def register() {
> > > val advertisedHostName =
> > > if(advertisedHost == null || advertisedHost.trim.isEmpty)
> > > InetAddress.getLocalHost.getCanonicalHostName
> > > else
> > > advertisedHost
> > > val jmxPort =
> > > System.getProperty("com.sun.management.jmxremote.port", "-1").toInt
> > > ZkUtils.registerBrokerInZk(zkClient, brokerId, advertisedHostName,
> > > advertisedPort, zkSessionTimeoutMs, jmxPort)
> > > }
> > >
> > > So never mind :)
> > >
> > >
> > > On Fri, Oct 17, 2014 at 6:36 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> > >> Hmm, yes, actually I don't think I actually understand the issue.
> Basically
> > >> as I understand it we do InetAddress.getLocalHost.getHostAddress
> which on
> > >> AWS picks the wrong hostname/ip and then the producer can't connect.
> People
> > >> eventually find this FAQ, but I was hoping there was a more automatic
> way
> > >> since everyone is on AWS these days. Maybe getCanonicalHostName would
> fix
> > >> it?
> > >>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers
> > >> ?
> > >>
> > >> -Jay
> > >>
> > >> On Fri, Oct 17, 2014 at 3:19 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> > >>
> > >>> In #2, do you refer to advertising the "internal" hostname instead of
> > >>> the external one?
> > >>> In this case, will it be enough to use getCanonicalHostName (which
> > >>> uses a name service)?
> > >>>
> > >>> Note that I think the problem the blog reported (wrong name
> > >>> advertised) is somewhat orthogonal to the question of which interface
> > >>> we bind to (which should probably be the default interface).
> > >>>
> > >>> Gwen
> > >>>
> > >>> On Fri, Oct 17, 2014 at 5:28 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> > >>> > This guy documented a few struggles getting going with Kafka. Not
> sure if
> > >>> > there is anything we can do to make it better?
> > >>> > http://ispyker.blogspot.com/2014/10/kafka-part-1.html
> > >>> >
> > >>> > 1. Would be great to figure out the apache/gradle thing.
> > >>> > 2. The problem of having Kafka advertise localhost on AWS is really
> > >>> common.
> > >>> > I was thinking one possible solution for this would be to get all
> the
> > >>> > interfaces and prefer non-localhost interfaces if they exist.
> > >>> >
> > >>> > -Jay
> > >>>
>
Re: blog with some "out of the box" pains
Posted by Ewen Cheslack-Postava <me...@ewencp.org>.
The first issue he runs into is one I also find frustrating -- with
cloud providers pushing SSDs, you have to use a pretty large instance
type to get a reasonable test setup. I'm not sure if he couldn't launch
an older type like m1.large (I think some newer AWS accounts aren't able
to) or if he just didn't see it as an option since they are hidden by
default. Even the largest general purpose instance types are pretty
wimpy wrt storage, only 80GB local instance storage.
The hostname issues are a well known pain point and unfortunately there
aren't any great solutions that aren't EC2-specific. Here's a quick run
down:
* None of the images for popular distros on EC2 will auto-set the
hostname beyond what EC2 already sets up (which isn't publicly
routable). The following details might explain why they can't. For
example, a recent Ubuntu image gives:
ubuntu@ip-172-30-2-76:~$ hostname
ip-172-30-2-76
ubuntu@ip-172-30-2-76:~$ cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
--- cut irrelevant pieces ---
* Sometimes the hostname is set, but isn't useful. For example, in this
Ubuntu image, the hostname is set to "ip-[ip-address-]", but that isn't
routable, so generates really irritating behavior. Running on the server
itself (which is running in a VPC, see below for more details):
scala> InetAddress.getLocalHost
java.net.UnknownHostException: ip-172-30-2-76: ip-172-30-2-76: Name or
service not known
at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
at .<init>(<console>:9)
at .<clinit>(<console>)
at .<init>(<console>:11)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:704)
at
scala.tools.nsc.interpreter.IMain$Request$$anonfun$14.apply(IMain.scala:920)
at
scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: ip-172-30-2-76: Name or
service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at
java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
... 14 more
* As described in a bunch of places, the only reliable way to get public
DNS info is through EC2's own instance metadata API:
https://forums.aws.amazon.com/thread.jspa?threadID=77788 For example:
curl -s http://169.254.169.254/latest/meta-data/public-hostname
might give something like:
ec2-203-0-113-25.compute-1.amazonaws.com
* But you may not even *have* a public DNS hostname. If you launch in a
VPC, you'll only get one if you set the VPC to generate them (and I'm
pretty sure the default is to not create them):
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html The
output of the curl call above will just be empty.
* AWS is pretty aggressively trying to move away from EC2-Classic (i.e.
non-VPC instances), so most new instances will end up in VPCs unless you
are working in a grandfathered account + AZ. If VPC without public DNS
is the default, we'll have to carefully guide new users in generating a
setup that works properly if we try to use hostnames.
* Even if you try moving the IP addresses, you still have to deal with
VPCs. You can't directly get your public IP address without accessing
something outside the host since you're in a VPC. You need to use the
instance metadata API to look it up, i.e.,
curl -s http://169.254.169.254/latest/meta-data/public-ipv4
* And yet another problem with IPs: unless you use an elastic IP, you're
not guaranteed they'll be stable:
Auto-assign Public IP
Requests a public IP address from Amazon's public IP address pool,
to make your instance reachable from the Internet. In most cases, the
public IP address is associated with the instance until it’s stopped
or
terminated, after which it’s no longer available for you to use. If
you
require a persistent public IP address that you can associate and
disassociate at will, use an Elastic IP address (EIP) instead. You can
allocate your own EIP, and associate it to your instance after launch.
I know Spark had some similar issues -- using their (very convenient!)
ec2 script, you still ended up with some stuff in their web interface
that linked to internal addresses such that the links wouldn't work. I'm
not sure if they have figured out a decent work around. But as you can
see from the above, it's unlikely you can use generic approaches to get
the info we need -- it'll need to be platform specific, which probably
means it's better to determine it outside the main Kafka code and
provide it via advertised.host.name.
-Ewen
On Fri, Oct 17, 2014, at 05:11 PM, Gwen Shapira wrote:
> Basically, the issue (or at least one of very many possible network
> issues...) is that the server has "localhost" hardcoded as its
> canonical name in /etc/hosts:
>
> [root@Billc-cent70x64 ~]# cat /etc/hosts
> 127.0.0.1 localhost localhost.localdomain localhost4
> localhost4.localdomain4 Billc-cent70x64
> ::1 localhost localhost.localdomain localhost6
> localhost6.localdomain6
>
> Unfortunately a very common default for RedHat and Centos machines.
>
> As the blog mentions, a good solution (other than instructing Kafka on
> the right name to advertise) is to add the correct IP and hostname to
> /etc/hosts. We may want to add this option to the FAQ.
>
> Gwen
>
>
>
>
> On Fri, Oct 17, 2014 at 7:56 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> > It looks like we are using canonical hostname:
> >
> > def register() {
> > val advertisedHostName =
> > if(advertisedHost == null || advertisedHost.trim.isEmpty)
> > InetAddress.getLocalHost.getCanonicalHostName
> > else
> > advertisedHost
> > val jmxPort =
> > System.getProperty("com.sun.management.jmxremote.port", "-1").toInt
> > ZkUtils.registerBrokerInZk(zkClient, brokerId, advertisedHostName,
> > advertisedPort, zkSessionTimeoutMs, jmxPort)
> > }
> >
> > So never mind :)
> >
> >
> > On Fri, Oct 17, 2014 at 6:36 PM, Jay Kreps <ja...@gmail.com> wrote:
> >> Hmm, yes, actually I don't think I actually understand the issue. Basically
> >> as I understand it we do InetAddress.getLocalHost.getHostAddress which on
> >> AWS picks the wrong hostname/ip and then the producer can't connect. People
> >> eventually find this FAQ, but I was hoping there was a more automatic way
> >> since everyone is on AWS these days. Maybe getCanonicalHostName would fix
> >> it?
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers
> >> ?
> >>
> >> -Jay
> >>
> >> On Fri, Oct 17, 2014 at 3:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> >>
> >>> In #2, do you refer to advertising the "internal" hostname instead of
> >>> the external one?
> >>> In this case, will it be enough to use getCanonicalHostName (which
> >>> uses a name service)?
> >>>
> >>> Note that I think the problem the blog reported (wrong name
> >>> advertised) is somewhat orthogonal to the question of which interface
> >>> we bind to (which should probably be the default interface).
> >>>
> >>> Gwen
> >>>
> >>> On Fri, Oct 17, 2014 at 5:28 PM, Jay Kreps <ja...@gmail.com> wrote:
> >>> > This guy documented a few struggles getting going with Kafka. Not sure if
> >>> > there is anything we can do to make it better?
> >>> > http://ispyker.blogspot.com/2014/10/kafka-part-1.html
> >>> >
> >>> > 1. Would be great to figure out the apache/gradle thing.
> >>> > 2. The problem of having Kafka advertise localhost on AWS is really
> >>> common.
> >>> > I was thinking one possible solution for this would be to get all the
> >>> > interfaces and prefer non-localhost interfaces if they exist.
> >>> >
> >>> > -Jay
> >>>
Re: blog with some "out of the box" pains
Posted by Gwen Shapira <gs...@cloudera.com>.
Basically, the issue (or at least one of very many possible network
issues...) is that the server has "localhost" hardcoded as its
canonical name in /etc/hosts:
[root@Billc-cent70x64 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4
localhost4.localdomain4 Billc-cent70x64
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Unfortunately a very common default for RedHat and Centos machines.
As the blog mentions, a good solution (other than instructing Kafka on
the right name to advertise) is to add the correct IP and hostname to
/etc/hosts. We may want to add this option to the FAQ.
Gwen
On Fri, Oct 17, 2014 at 7:56 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> It looks like we are using canonical hostname:
>
> def register() {
> val advertisedHostName =
> if(advertisedHost == null || advertisedHost.trim.isEmpty)
> InetAddress.getLocalHost.getCanonicalHostName
> else
> advertisedHost
> val jmxPort =
> System.getProperty("com.sun.management.jmxremote.port", "-1").toInt
> ZkUtils.registerBrokerInZk(zkClient, brokerId, advertisedHostName,
> advertisedPort, zkSessionTimeoutMs, jmxPort)
> }
>
> So never mind :)
>
>
> On Fri, Oct 17, 2014 at 6:36 PM, Jay Kreps <ja...@gmail.com> wrote:
>> Hmm, yes, actually I don't think I actually understand the issue. Basically
>> as I understand it we do InetAddress.getLocalHost.getHostAddress which on
>> AWS picks the wrong hostname/ip and then the producer can't connect. People
>> eventually find this FAQ, but I was hoping there was a more automatic way
>> since everyone is on AWS these days. Maybe getCanonicalHostName would fix
>> it?
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers
>> ?
>>
>> -Jay
>>
>> On Fri, Oct 17, 2014 at 3:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>>
>>> In #2, do you refer to advertising the "internal" hostname instead of
>>> the external one?
>>> In this case, will it be enough to use getCanonicalHostName (which
>>> uses a name service)?
>>>
>>> Note that I think the problem the blog reported (wrong name
>>> advertised) is somewhat orthogonal to the question of which interface
>>> we bind to (which should probably be the default interface).
>>>
>>> Gwen
>>>
>>> On Fri, Oct 17, 2014 at 5:28 PM, Jay Kreps <ja...@gmail.com> wrote:
>>> > This guy documented a few struggles getting going with Kafka. Not sure if
>>> > there is anything we can do to make it better?
>>> > http://ispyker.blogspot.com/2014/10/kafka-part-1.html
>>> >
>>> > 1. Would be great to figure out the apache/gradle thing.
>>> > 2. The problem of having Kafka advertise localhost on AWS is really
>>> common.
>>> > I was thinking one possible solution for this would be to get all the
>>> > interfaces and prefer non-localhost interfaces if they exist.
>>> >
>>> > -Jay
>>>
Re: blog with some "out of the box" pains
Posted by Gwen Shapira <gs...@cloudera.com>.
It looks like we are using canonical hostname:
def register() {
val advertisedHostName =
if(advertisedHost == null || advertisedHost.trim.isEmpty)
InetAddress.getLocalHost.getCanonicalHostName
else
advertisedHost
val jmxPort =
System.getProperty("com.sun.management.jmxremote.port", "-1").toInt
ZkUtils.registerBrokerInZk(zkClient, brokerId, advertisedHostName,
advertisedPort, zkSessionTimeoutMs, jmxPort)
}
So never mind :)
On Fri, Oct 17, 2014 at 6:36 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hmm, yes, actually I don't think I actually understand the issue. Basically
> as I understand it we do InetAddress.getLocalHost.getHostAddress which on
> AWS picks the wrong hostname/ip and then the producer can't connect. People
> eventually find this FAQ, but I was hoping there was a more automatic way
> since everyone is on AWS these days. Maybe getCanonicalHostName would fix
> it?
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers
> ?
>
> -Jay
>
> On Fri, Oct 17, 2014 at 3:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
>> In #2, do you refer to advertising the "internal" hostname instead of
>> the external one?
>> In this case, will it be enough to use getCanonicalHostName (which
>> uses a name service)?
>>
>> Note that I think the problem the blog reported (wrong name
>> advertised) is somewhat orthogonal to the question of which interface
>> we bind to (which should probably be the default interface).
>>
>> Gwen
>>
>> On Fri, Oct 17, 2014 at 5:28 PM, Jay Kreps <ja...@gmail.com> wrote:
>> > This guy documented a few struggles getting going with Kafka. Not sure if
>> > there is anything we can do to make it better?
>> > http://ispyker.blogspot.com/2014/10/kafka-part-1.html
>> >
>> > 1. Would be great to figure out the apache/gradle thing.
>> > 2. The problem of having Kafka advertise localhost on AWS is really
>> common.
>> > I was thinking one possible solution for this would be to get all the
>> > interfaces and prefer non-localhost interfaces if they exist.
>> >
>> > -Jay
>>
Re: blog with some "out of the box" pains
Posted by Jay Kreps <ja...@gmail.com>.
Hmm, yes, actually I don't think I actually understand the issue. Basically
as I understand it we do InetAddress.getLocalHost.getHostAddress which on
AWS picks the wrong hostname/ip and then the producer can't connect. People
eventually find this FAQ, but I was hoping there was a more automatic way
since everyone is on AWS these days. Maybe getCanonicalHostName would fix
it?
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers
?
-Jay
On Fri, Oct 17, 2014 at 3:19 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> In #2, do you refer to advertising the "internal" hostname instead of
> the external one?
> In this case, will it be enough to use getCanonicalHostName (which
> uses a name service)?
>
> Note that I think the problem the blog reported (wrong name
> advertised) is somewhat orthogonal to the question of which interface
> we bind to (which should probably be the default interface).
>
> Gwen
>
> On Fri, Oct 17, 2014 at 5:28 PM, Jay Kreps <ja...@gmail.com> wrote:
> > This guy documented a few struggles getting going with Kafka. Not sure if
> > there is anything we can do to make it better?
> > http://ispyker.blogspot.com/2014/10/kafka-part-1.html
> >
> > 1. Would be great to figure out the apache/gradle thing.
> > 2. The problem of having Kafka advertise localhost on AWS is really
> common.
> > I was thinking one possible solution for this would be to get all the
> > interfaces and prefer non-localhost interfaces if they exist.
> >
> > -Jay
>
Re: blog with some "out of the box" pains
Posted by Gwen Shapira <gs...@cloudera.com>.
In #2, do you refer to advertising the "internal" hostname instead of
the external one?
In this case, will it be enough to use getCanonicalHostName (which
uses a name service)?
Note that I think the problem the blog reported (wrong name
advertised) is somewhat orthogonal to the question of which interface
we bind to (which should probably be the default interface).
Gwen
On Fri, Oct 17, 2014 at 5:28 PM, Jay Kreps <ja...@gmail.com> wrote:
> This guy documented a few struggles getting going with Kafka. Not sure if
> there is anything we can do to make it better?
> http://ispyker.blogspot.com/2014/10/kafka-part-1.html
>
> 1. Would be great to figure out the apache/gradle thing.
> 2. The problem of having Kafka advertise localhost on AWS is really common.
> I was thinking one possible solution for this would be to get all the
> interfaces and prefer non-localhost interfaces if they exist.
>
> -Jay