You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Aniket Bhatnagar <an...@gmail.com> on 2013/10/03 08:57:58 UTC
Metadata API returns localhost.localdomain for one of the brokers in EC2
I have installed 2 brokers on EC2. I also have a (scala) application that
receives data stream and pushes to kafka cluster. By co-incidence, a
(slightly heavier) EC2 instance is running both a kafka broker and the data
receiver application. I am noticing that all data receiver application
nodes that are not on the shared kafka + reciever app EC2 instance are
complaining for connect errors to localhost.localdomain:9092. Is this a
possible bug that results in Kafka detecting instance hostname
as localhost.localdomain instead of actual hostname?
Also, how do I fix this temporarily until a permanent fix is available?
Re: Metadata API returns localhost.localdomain for one of the brokers
in EC2
Posted by Jay Kreps <ja...@gmail.com>.
Is there something we could be doing in Kafka to avoid this problem?
-Jay
On Thu, Oct 3, 2013 at 8:29 AM, Rajasekar Elango <re...@salesforce.com>wrote:
> Hi Aniket,
>
> We had same issue it turns out that we need to make sure ip to hostname
> mapping should be correctly configured in /etc/hosts file.
>
> For eg: If you had something like
> 127.0.0.1 localhost localhost
>
> as first line in /etc/hosts file, you will get his error. To fix we need to
> add correct ip to hostname mapping as first line in /etc/hosts file.
>
> Hope this helps.
>
> Thanks,
> Raja.
>
>
>
> On Thu, Oct 3, 2013 at 11:02 AM, Aniket Bhatnagar <
> aniket.bhatnagar@gmail.com> wrote:
>
> > Thanks Jun and David.
> >
> > I think the FAQ mentions why it's not possible to connect to broker from
> > outside. In my case, all servers (producers and brokers) are in the same
> > VPC. Call to InetAddress.getLocalHost.getHostAddress should return an
> > internal IP to which producers should be able to connect. The issue seems
> > to be that the call to InetAddress.getLocalHost.getHostAddress returns
> > localhost.localdomain
> > and that too on just 1 broker. Any ideas on why that could be happening?
> >
> > I can configure host.name property in broker config but its slightly
> > painful. I am curious to know what can cause
> > InetAddress.getLocalHost.getHostAddress
> > to return loopback addresses like localhost.localdomain so that other
> users
> > know why they really have to setup host.name in EC2 even if both
> producers
> > and consumers are in same VPC.
> >
> >
> > On 3 October 2013 19:56, Jun Rao <ju...@gmail.com> wrote:
> >
> > > There is an FAQ too.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Oct 3, 2013 at 6:43 AM, David Arthur <mu...@gmail.com> wrote:
> > >
> > > > You can configure the hostname for the broker with the "host.name"
> > > > property in the broker's config (server.properties?). If you don't
> > > specify
> > > > one here, then all interfaces will be bound to and one will be chosen
> > to
> > > > get published via ZooKeeper (what the metadata API is reading)
> > > >
> > > > See: http://kafka.apache.org/**documentation.html#**brokerconfigs<
> > > http://kafka.apache.org/documentation.html#brokerconfigs>
> > > >
> > > > -David
> > > >
> > > >
> > > > On 10/3/13 2:57 AM, Aniket Bhatnagar wrote:
> > > >
> > > >> I have installed 2 brokers on EC2. I also have a (scala) application
> > > that
> > > >> receives data stream and pushes to kafka cluster. By co-incidence, a
> > > >> (slightly heavier) EC2 instance is running both a kafka broker and
> the
> > > >> data
> > > >> receiver application. I am noticing that all data receiver
> application
> > > >> nodes that are not on the shared kafka + reciever app EC2 instance
> are
> > > >> complaining for connect errors to localhost.localdomain:9092. Is
> this
> > a
> > > >> possible bug that results in Kafka detecting instance hostname
> > > >> as localhost.localdomain instead of actual hostname?
> > > >>
> > > >> Also, how do I fix this temporarily until a permanent fix is
> > available?
> > > >>
> > > >>
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Raja.
>
Re: Metadata API returns localhost.localdomain for one of the brokers
in EC2
Posted by Rajasekar Elango <re...@salesforce.com>.
Hi Aniket,
We had same issue it turns out that we need to make sure ip to hostname
mapping should be correctly configured in /etc/hosts file.
For eg: If you had something like
127.0.0.1 localhost localhost
as first line in /etc/hosts file, you will get his error. To fix we need to
add correct ip to hostname mapping as first line in /etc/hosts file.
Hope this helps.
Thanks,
Raja.
On Thu, Oct 3, 2013 at 11:02 AM, Aniket Bhatnagar <
aniket.bhatnagar@gmail.com> wrote:
> Thanks Jun and David.
>
> I think the FAQ mentions why it's not possible to connect to broker from
> outside. In my case, all servers (producers and brokers) are in the same
> VPC. Call to InetAddress.getLocalHost.getHostAddress should return an
> internal IP to which producers should be able to connect. The issue seems
> to be that the call to InetAddress.getLocalHost.getHostAddress returns
> localhost.localdomain
> and that too on just 1 broker. Any ideas on why that could be happening?
>
> I can configure host.name property in broker config but its slightly
> painful. I am curious to know what can cause
> InetAddress.getLocalHost.getHostAddress
> to return loopback addresses like localhost.localdomain so that other users
> know why they really have to setup host.name in EC2 even if both producers
> and consumers are in same VPC.
>
>
> On 3 October 2013 19:56, Jun Rao <ju...@gmail.com> wrote:
>
> > There is an FAQ too.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Oct 3, 2013 at 6:43 AM, David Arthur <mu...@gmail.com> wrote:
> >
> > > You can configure the hostname for the broker with the "host.name"
> > > property in the broker's config (server.properties?). If you don't
> > specify
> > > one here, then all interfaces will be bound to and one will be chosen
> to
> > > get published via ZooKeeper (what the metadata API is reading)
> > >
> > > See: http://kafka.apache.org/**documentation.html#**brokerconfigs<
> > http://kafka.apache.org/documentation.html#brokerconfigs>
> > >
> > > -David
> > >
> > >
> > > On 10/3/13 2:57 AM, Aniket Bhatnagar wrote:
> > >
> > >> I have installed 2 brokers on EC2. I also have a (scala) application
> > that
> > >> receives data stream and pushes to kafka cluster. By co-incidence, a
> > >> (slightly heavier) EC2 instance is running both a kafka broker and the
> > >> data
> > >> receiver application. I am noticing that all data receiver application
> > >> nodes that are not on the shared kafka + reciever app EC2 instance are
> > >> complaining for connect errors to localhost.localdomain:9092. Is this
> a
> > >> possible bug that results in Kafka detecting instance hostname
> > >> as localhost.localdomain instead of actual hostname?
> > >>
> > >> Also, how do I fix this temporarily until a permanent fix is
> available?
> > >>
> > >>
> > >
> >
>
--
Thanks,
Raja.
Re: Metadata API returns localhost.localdomain for one of the brokers
in EC2
Posted by Aniket Bhatnagar <an...@gmail.com>.
Thanks Jun and David.
I think the FAQ mentions why it's not possible to connect to broker from
outside. In my case, all servers (producers and brokers) are in the same
VPC. Call to InetAddress.getLocalHost.getHostAddress should return an
internal IP to which producers should be able to connect. The issue seems
to be that the call to InetAddress.getLocalHost.getHostAddress returns
localhost.localdomain
and that too on just 1 broker. Any ideas on why that could be happening?
I can configure host.name property in broker config but its slightly
painful. I am curious to know what can cause
InetAddress.getLocalHost.getHostAddress
to return loopback addresses like localhost.localdomain so that other users
know why they really have to setup host.name in EC2 even if both producers
and consumers are in same VPC.
On 3 October 2013 19:56, Jun Rao <ju...@gmail.com> wrote:
> There is an FAQ too.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F
>
> Thanks,
>
> Jun
>
>
> On Thu, Oct 3, 2013 at 6:43 AM, David Arthur <mu...@gmail.com> wrote:
>
> > You can configure the hostname for the broker with the "host.name"
> > property in the broker's config (server.properties?). If you don't
> specify
> > one here, then all interfaces will be bound to and one will be chosen to
> > get published via ZooKeeper (what the metadata API is reading)
> >
> > See: http://kafka.apache.org/**documentation.html#**brokerconfigs<
> http://kafka.apache.org/documentation.html#brokerconfigs>
> >
> > -David
> >
> >
> > On 10/3/13 2:57 AM, Aniket Bhatnagar wrote:
> >
> >> I have installed 2 brokers on EC2. I also have a (scala) application
> that
> >> receives data stream and pushes to kafka cluster. By co-incidence, a
> >> (slightly heavier) EC2 instance is running both a kafka broker and the
> >> data
> >> receiver application. I am noticing that all data receiver application
> >> nodes that are not on the shared kafka + reciever app EC2 instance are
> >> complaining for connect errors to localhost.localdomain:9092. Is this a
> >> possible bug that results in Kafka detecting instance hostname
> >> as localhost.localdomain instead of actual hostname?
> >>
> >> Also, how do I fix this temporarily until a permanent fix is available?
> >>
> >>
> >
>
Re: Metadata API returns localhost.localdomain for one of the brokers
in EC2
Posted by Jun Rao <ju...@gmail.com>.
There is an FAQ too.
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2%2Cwhycan%27tmyhighlevelconsumersconnecttothebrokers%3F
Thanks,
Jun
On Thu, Oct 3, 2013 at 6:43 AM, David Arthur <mu...@gmail.com> wrote:
> You can configure the hostname for the broker with the "host.name"
> property in the broker's config (server.properties?). If you don't specify
> one here, then all interfaces will be bound to and one will be chosen to
> get published via ZooKeeper (what the metadata API is reading)
>
> See: http://kafka.apache.org/**documentation.html#**brokerconfigs<http://kafka.apache.org/documentation.html#brokerconfigs>
>
> -David
>
>
> On 10/3/13 2:57 AM, Aniket Bhatnagar wrote:
>
>> I have installed 2 brokers on EC2. I also have a (scala) application that
>> receives data stream and pushes to kafka cluster. By co-incidence, a
>> (slightly heavier) EC2 instance is running both a kafka broker and the
>> data
>> receiver application. I am noticing that all data receiver application
>> nodes that are not on the shared kafka + reciever app EC2 instance are
>> complaining for connect errors to localhost.localdomain:9092. Is this a
>> possible bug that results in Kafka detecting instance hostname
>> as localhost.localdomain instead of actual hostname?
>>
>> Also, how do I fix this temporarily until a permanent fix is available?
>>
>>
>
Re: Metadata API returns localhost.localdomain for one of the brokers
in EC2
Posted by David Arthur <mu...@gmail.com>.
You can configure the hostname for the broker with the "host.name"
property in the broker's config (server.properties?). If you don't
specify one here, then all interfaces will be bound to and one will be
chosen to get published via ZooKeeper (what the metadata API is reading)
See: http://kafka.apache.org/documentation.html#brokerconfigs
-David
On 10/3/13 2:57 AM, Aniket Bhatnagar wrote:
> I have installed 2 brokers on EC2. I also have a (scala) application that
> receives data stream and pushes to kafka cluster. By co-incidence, a
> (slightly heavier) EC2 instance is running both a kafka broker and the data
> receiver application. I am noticing that all data receiver application
> nodes that are not on the shared kafka + reciever app EC2 instance are
> complaining for connect errors to localhost.localdomain:9092. Is this a
> possible bug that results in Kafka detecting instance hostname
> as localhost.localdomain instead of actual hostname?
>
> Also, how do I fix this temporarily until a permanent fix is available?
>