You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Shekar Tippur <ct...@gmail.com> on 2016/08/04 00:56:38 UTC
Samza yarn job - cannot bind to local host
I am trying to submit a Samza job to yarn and I get a error:
Exception in thread "main" java.io.IOException: Cannot bind to URL
[rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailableException
[Root exception is java.rmi.ConnectException: Connection refused to host:
localhost; nested exception is:
java.net.ConnectException: Connection refused]
at
javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:827)
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:432)
at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:89)
at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:43)
at
org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:87)
at
org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
Caused by: javax.naming.ServiceUnavailableException [Root exception is
java.rmi.ConnectException: Connection refused to host: localhost; nested
exception is:
java.net.ConnectException: Connection refused]
at
com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:147)
at
com.sun.jndi.toolkit.url.GenericURLContext.bind(GenericURLContext.java:228)
at javax.naming.InitialContext.bind(InitialContext.java:425)
at
javax.management.remote.rmi.RMIConnectorServer.bind(RMIConnectorServer.java:644)
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:427)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host:
localhost; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at
sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
at
com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:141)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 13 more
Any idea on what could be the issue?
- Shekar
Re: Samza yarn job - cannot bind to local host
Posted by Jacob Maes <ja...@gmail.com>.
Hey Shekar,
Yes, https://issues.apache.org/jira/browse/SAMZA-825 seems to fit the bill.
It hasn't been high priority because most users run samza in YARN or some
other multi-tenant environment s.t. the ephemeral ports are ideal. However
with the upcoming standalone support, this makes more sense.
-Jake
On Wed, Aug 10, 2016 at 10:37 AM, Shekar Tippur <ct...@gmail.com> wrote:
> Just following on the same thread, is there a feature/bug to force the
> jmxrmi port? This would be helpful to get the container metrics.
>
> - Shekar
>
> On Sat, Aug 6, 2016 at 6:14 PM, Shekar Tippur <ct...@gmail.com> wrote:
>
> > Thanks a lot guys. It was a system problem. /etc/hosts file had wrong
> > permissions. localhost was being resolved to a weird ip for certain users
> > via DNS.
> > - Shekar
> >
> > On Fri, Aug 5, 2016 at 11:27 AM, Jacob Maes <ja...@gmail.com>
> wrote:
> >
> >> Hey Shekar,
> >>
> >> There's currently no way to force the port or disable the JMXServer.
> >>
> >> We've seen port conflicts (fixed in 10.1) and connection issues due to
> VPN
> >> but no connection issue for other reasons.
> >>
> >> While googling that exception, I found a couple cases without clear
> >> resolutions, but they were both on Windows machines. Are your servers
> >> running Windows?
> >>
> >> Ultimately it seems that there is something atypical about your
> >> environment
> >> compared to how Samza is commonly run. If we can determine what that is,
> >> hopefully that'll lead to a fix in either the environment or Samza. So
> any
> >> other details about your cluster you can provide would be useful.
> >>
> >> -Jake
> >>
> >> On Fri, Aug 5, 2016 at 7:33 AM, Shekar Tippur <ct...@gmail.com>
> wrote:
> >>
> >> > Any pointers on this please. I am completely blocked.
> >> >
> >> > - Shekar
> >> >
> >> > On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur <ct...@gmail.com>
> >> wrote:
> >> >
> >> > > This server is not connected to vpn.
> >> > >
> >> > > - Shekar
> >> > >
> >> >
> >>
> >
> >
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
Just following on the same thread, is there a feature/bug to force the
jmxrmi port? This would be helpful to get the container metrics.
- Shekar
On Sat, Aug 6, 2016 at 6:14 PM, Shekar Tippur <ct...@gmail.com> wrote:
> Thanks a lot guys. It was a system problem. /etc/hosts file had wrong
> permissions. localhost was being resolved to a weird ip for certain users
> via DNS.
> - Shekar
>
> On Fri, Aug 5, 2016 at 11:27 AM, Jacob Maes <ja...@gmail.com> wrote:
>
>> Hey Shekar,
>>
>> There's currently no way to force the port or disable the JMXServer.
>>
>> We've seen port conflicts (fixed in 10.1) and connection issues due to VPN
>> but no connection issue for other reasons.
>>
>> While googling that exception, I found a couple cases without clear
>> resolutions, but they were both on Windows machines. Are your servers
>> running Windows?
>>
>> Ultimately it seems that there is something atypical about your
>> environment
>> compared to how Samza is commonly run. If we can determine what that is,
>> hopefully that'll lead to a fix in either the environment or Samza. So any
>> other details about your cluster you can provide would be useful.
>>
>> -Jake
>>
>> On Fri, Aug 5, 2016 at 7:33 AM, Shekar Tippur <ct...@gmail.com> wrote:
>>
>> > Any pointers on this please. I am completely blocked.
>> >
>> > - Shekar
>> >
>> > On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur <ct...@gmail.com>
>> wrote:
>> >
>> > > This server is not connected to vpn.
>> > >
>> > > - Shekar
>> > >
>> >
>>
>
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
Thanks a lot guys. It was a system problem. /etc/hosts file had wrong
permissions. localhost was being resolved to a weird ip for certain users
via DNS.
- Shekar
On Fri, Aug 5, 2016 at 11:27 AM, Jacob Maes <ja...@gmail.com> wrote:
> Hey Shekar,
>
> There's currently no way to force the port or disable the JMXServer.
>
> We've seen port conflicts (fixed in 10.1) and connection issues due to VPN
> but no connection issue for other reasons.
>
> While googling that exception, I found a couple cases without clear
> resolutions, but they were both on Windows machines. Are your servers
> running Windows?
>
> Ultimately it seems that there is something atypical about your environment
> compared to how Samza is commonly run. If we can determine what that is,
> hopefully that'll lead to a fix in either the environment or Samza. So any
> other details about your cluster you can provide would be useful.
>
> -Jake
>
> On Fri, Aug 5, 2016 at 7:33 AM, Shekar Tippur <ct...@gmail.com> wrote:
>
> > Any pointers on this please. I am completely blocked.
> >
> > - Shekar
> >
> > On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur <ct...@gmail.com> wrote:
> >
> > > This server is not connected to vpn.
> > >
> > > - Shekar
> > >
> >
>
Re: Samza yarn job - cannot bind to local host
Posted by Jacob Maes <ja...@gmail.com>.
Hey Shekar,
There's currently no way to force the port or disable the JMXServer.
We've seen port conflicts (fixed in 10.1) and connection issues due to VPN
but no connection issue for other reasons.
While googling that exception, I found a couple cases without clear
resolutions, but they were both on Windows machines. Are your servers
running Windows?
Ultimately it seems that there is something atypical about your environment
compared to how Samza is commonly run. If we can determine what that is,
hopefully that'll lead to a fix in either the environment or Samza. So any
other details about your cluster you can provide would be useful.
-Jake
On Fri, Aug 5, 2016 at 7:33 AM, Shekar Tippur <ct...@gmail.com> wrote:
> Any pointers on this please. I am completely blocked.
>
> - Shekar
>
> On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur <ct...@gmail.com> wrote:
>
> > This server is not connected to vpn.
> >
> > - Shekar
> >
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
Any pointers on this please. I am completely blocked.
- Shekar
On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur <ct...@gmail.com> wrote:
> This server is not connected to vpn.
>
> - Shekar
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
This server is not connected to vpn.
- Shekar
Re: Samza yarn job - cannot bind to local host
Posted by Jacob Maes <ja...@gmail.com>.
Hey Shekar,
Are your YARN hosts connected to a VPN by chance?
The only time I've seen problems with the localhost was when I ran unit
tests on my Mac while connected to VPN. Apparently, its related to a bug in
Java http://bugs.java.com/view_bug.do?bug_id=7180557
-Jake
On Thu, Aug 4, 2016 at 4:06 PM, Shekar Tippur <ct...@gmail.com> wrote:
> Is there anyway we can disable this?
>
> - Shekar
>
> On Thu, Aug 4, 2016 at 4:02 PM, Shekar Tippur <ct...@gmail.com> wrote:
>
> > Yi,
> >
> > I can do a curl or telnet to localhost but it is hard to simulate as it
> > spawns random ports.
> > The job fails.
> >
> > Here is the application log:
> >
> > 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got container id:
> > container_1470270235966_0016_02_000001
> >
> > 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got app attempt id:
> > appattempt_1470270235966_0016_000002
> >
> > 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager host:
> > sprdargas102
> >
> > 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager port: 40531
> >
> > 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager http port:
> 8042
> >
> > 2016-08-04 20:40:57 SamzaAppMaster$ [INFO] got coordinator system config:
> > {systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,
> > sprdargas103:2181,sprdargas104:2181, systems.kafka.samza.factory=
> > org.apache.samza.system.kafka.KafkaSystemFactory,
> > job.coordinator.system=kafka, systems.kafka.streams.metrics.
> samza.msg.serde=metrics,
> > job.id=1, systems.kafka.producer.producer.type=sync,
> > systems.kafka.producer.metadata.broker.list=sprdargas102:6667,
> sprdargas103:6667,
> > systems.kafka.producer.batch.num.messages=1,
> > systems.kafka.samza.msg.serde=json, systems.kafka.consumer.auto.
> offset.reset=largest,
> > systems.kafka.streams.PageViewEvent.samza.msg.serde=json,
> > systems.kafka.streams.ArgosRaw.samza.msg.serde=json, job.name=Argos,
> > systems.kafka.producer.bootstrap.servers=sprdargas102:6667,
> > sprdargas103:6667}
> >
> > 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> > configuration contains 'metadata.broker.list'. This configuration is
> > deprecated . Samza has been upgraded to use Kafka's new producer API.
> > Please update your configurations based on the documentation at
> > http://kafka.apache.org/documentation.html#newproducerconfigs
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> auto.offset.reset
> > is overridden to largest
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> > overridden to samza_admin-Argos-1-1470343257174-0
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> > overridden to undefined-samza-consumer-group-427db503-329d-4201-87da-
> > eed83f9f1390
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> zookeeper.connect
> > is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
> >
> > 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> > configuration contains 'metadata.broker.list'. This configuration is
> > deprecated . Samza has been upgraded to use Kafka's new producer API.
> > Please update your configurations based on the documentation at
> > http://kafka.apache.org/documentation.html#newproducerconfigs
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> auto.offset.reset
> > is overridden to largest
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> > overridden to samza_consumer-Argos-1-1470343257242-1
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> > overridden to undefined-samza-consumer-group-aa7250b3-ffa6-4caa-8952-
> > 6aaaa8c275f1
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> zookeeper.connect
> > is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
> >
> > 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> > configuration contains 'metadata.broker.list'. This configuration is
> > deprecated . Samza has been upgraded to use Kafka's new producer API.
> > Please update your configurations based on the documentation at
> > http://kafka.apache.org/documentation.html#newproducerconfigs
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> auto.offset.reset
> > is overridden to largest
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> > overridden to samza_admin-Argos-1-1470343257270-2
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> > overridden to undefined-samza-consumer-group-00ed785c-1ee2-41cd-82ec-
> > 78c7a5e4f404
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> zookeeper.connect
> > is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
> >
> > 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> > configuration contains 'metadata.broker.list'. This configuration is
> > deprecated . Samza has been upgraded to use Kafka's new producer API.
> > Please update your configurations based on the documentation at
> > http://kafka.apache.org/documentation.html#newproducerconfigs
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> auto.offset.reset
> > is overridden to largest
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> > overridden to samza_admin-Argos-1-1470343257283-3
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> > overridden to undefined-samza-consumer-group-d75f6419-4985-45d1-aeed-
> > 18351ca05be5
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> zookeeper.connect
> > is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
> >
> > 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> > configuration contains 'metadata.broker.list'. This configuration is
> > deprecated . Samza has been upgraded to use Kafka's new producer API.
> > Please update your configurations based on the documentation at
> > http://kafka.apache.org/documentation.html#newproducerconfigs
> >
> > 2016-08-04 20:40:57 JobCoordinator$ [INFO] Registering coordinator system
> > stream.
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> > overridden to samza_admin-Argos-1-1470343257174-0
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> > metadata.broker.list is overridden to sprdargas102:6667,
> sprdargas103:6667
> >
> > 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> > request.timeout.ms is overridden to 30000
> >
> > 2016-08-04 20:40:57 ClientUtils$ [INFO] Fetching metadata from broker
> > id:0,host:sprdargas102,port:6667 with correlation id 0 for 1 topic(s)
> > Set(__samza_coordinator_Argos_1)
> >
> > 2016-08-04 20:40:57 SyncProducer [INFO] Connected to sprdargas102:6667
> for
> > producing
> >
> > 2016-08-04 20:40:57 SyncProducer [INFO] Disconnecting from
> > sprdargas102:6667
> >
> > 2016-08-04 20:40:57 KafkaSystemAdmin$ [INFO] Got metadata:
> > Map(__samza_coordinator_Argos_1 -> SystemStreamMetadata
> > [streamName=__samza_coordinator_Argos_1, partitionMetadata={Partition
> > [partition=0]=SystemStreamPartitionMetadata [oldestOffset=0,
> > newestOffset=1269, upcomingOffset=1270]}])
> >
> > 2016-08-04 20:40:57 CoordinatorStreamSystemConsumer [INFO] Starting
> > coordinator stream system consumer.
> >
> > 2016-08-04 20:40:57 KafkaSystemConsumer [INFO] Refreshing brokers for:
> > Map([__samza_coordinator_Argos_1,0] -> 0)
> >
> > 2016-08-04 20:40:57 BrokerProxy [INFO] Creating new SimpleConsumer for
> > host sprdargas102:6667 for system kafka
> >
> > 2016-08-04 20:40:57 GetOffset [INFO] Validating offset 0 for topic and
> > partition [__samza_coordinator_Argos_1,0]
> >
> > 2016-08-04 20:40:57 GetOffset [INFO] Able to successfully read from
> offset
> > 0 for topic and partition [__samza_coordinator_Argos_1,0]. Using it to
> > instantiate consumer.
> >
> > 2016-08-04 20:40:57 BrokerProxy [INFO] Starting BrokerProxy for
> > sprdargas102:6667
> >
> > 2016-08-04 20:40:57 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> > configuration from coordinator stream.
> >
> > 2016-08-04 20:40:58 JobCoordinator$ [INFO] Got config:
> > {systems.kafka.samza.factory=org.apache.samza.system.kafka.
> KafkaSystemFactory,
> > system.redis=sprdargas104, task.drop.deserialization.errors=true,
> > serializers.registry.metrics.class=org.apache.samza.serializers.
> MetricsSnapshotSerdeFactory,
> > systems.kafka.streams.metrics.samza.msg.serde=metrics,
> > serializers.registry.string.class=org.apache.samza.serializers.
> StringSerdeFactory,
> > task.checkpoint.system=kafka, task.checkpoint.factory=org.
> > apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory,
> > task.ignored.exceptions=org.codehaus.jackson.JsonParseException,
> > systems.kafka.producer.metadata.broker.list=sprdargas102:6667,
> sprdargas103:6667,
> > systems.kafka.samza.msg.serde=json, metrics.reporters=snapshot,jmx,
> > systems.kafka.consumer.auto.offset.reset=largest, job.name=Argos,
> > metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.
> > MetricsSnapshotReporterFactory, systems.kafka.producer.
> bootstrap.servers=
> > sprdargas102:6667,sprdargas103:6667, task.checkpoint.replication.
> factor=2,
> > systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,
> > sprdargas103:2181,sprdargas104:2181, load.property=true,
> > job.coordinator.system=kafka, task.inputs=kafka.argos-raw,
> > systems.kafka.producer.producer.type=sync, task.outputs=argos-parser,
> > yarn.package.path=http://sprdargas102:8000/argos_aggregate.tar.gz,
> > job.factory.class=org.apache.samza.job.yarn.YarnJobFactory,
> > task.class=com.intuit.argos.task.ArgosParserStreamTask,
> > redis.wildcard.lookup=true, metrics.reporter.jmx.class=
> > org.apache.samza.metrics.reporter.JmxReporterFactory,
> > systems.kafka.producer.batch.num.messages=1, systems.kafka.streams.
> > PageViewEvent.samza.msg.serde=json, task.drop.serialization.errors=true,
> > systems.kafka.streams.ArgosRaw.samza.msg.serde=json,
> > serializers.registry.json.class=org.apache.samza.
> serializers.JsonSerdeFactory,
> > task.opts=-Dcom.sun.management.jmxremote -Dcom.sun.management.
> jmxremote.port=1099
> > -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.
> jmxremote.ssl=false
> > -Djava.rmi.server.hostname=127.0.0.1, metrics.reporter.snapshot.
> > stream=kafka.metrics}
> >
> > 2016-08-04 20:40:58 KafkaProducerConfig [WARN] Kafka producer
> > configuration contains 'metadata.broker.list'. This configuration is
> > deprecated . Samza has been upgraded to use Kafka's new producer API.
> > Please update your configurations based on the documentation at
> > http://kafka.apache.org/documentation.html#newproducerconfigs
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Verifying properties
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Property
> auto.offset.reset
> > is overridden to largest
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Property client.id is
> > overridden to samza_admin-Argos-1-1470343258775-5
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Property group.id is
> > overridden to undefined-samza-consumer-group-b18c1f75-fd2a-4eea-bb28-
> > 98d5a85fc3c8
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Property
> zookeeper.connect
> > is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
> >
> > 2016-08-04 20:40:58 JobConfig [INFO] job.coordinator.replication.factor
> > was not found. Using task.checkpoint.replication.factor=2 for
> coordinator
> > stream
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Verifying properties
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Property client.id is
> > overridden to samza_admin-Argos-1-1470343258775-5
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Property
> > metadata.broker.list is overridden to sprdargas102:6667,
> sprdargas103:6667
> >
> > 2016-08-04 20:40:58 VerifiableProperties [INFO] Property
> > request.timeout.ms is overridden to 30000
> >
> > 2016-08-04 20:40:58 ClientUtils$ [INFO] Fetching metadata from broker
> > id:1,host:sprdargas103,port:6667 with correlation id 0 for 1 topic(s)
> > Set(argos-raw)
> >
> > 2016-08-04 20:40:58 SyncProducer [INFO] Connected to sprdargas103:6667
> for
> > producing
> >
> > 2016-08-04 20:40:58 SyncProducer [INFO] Disconnecting from
> > sprdargas103:6667
> >
> > 2016-08-04 20:40:58 KafkaSystemAdmin$ [INFO] Got metadata: Map(argos-raw
> > -> SystemStreamMetadata [streamName=argos-raw,
> partitionMetadata={Partition
> > [partition=0]=SystemStreamPartitionMetadata [oldestOffset=0,
> > newestOffset=null, upcomingOffset=0]}])
> >
> > 2016-08-04 20:40:58 JobCoordinator$ [INFO] SystemStreamPartitionGrouper
> > org.apache.samza.container.grouper.stream.GroupByPartition@5674e1f2 has
> > grouped the SystemStreamPartitions into the following taskNames:
> >
> > 2016-08-04 20:40:58 CoordinatorStreamSystemProducer [INFO] Starting
> > coordinator stream producer.
> >
> > 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Coordinator
> > stream consumer already started
> >
> > 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> > configuration from coordinator stream.
> >
> > 2016-08-04 20:40:58 CoordinatorStreamSystemProducer [INFO] Coordinator
> > stream producer already started
> >
> > 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Coordinator
> > stream consumer already started
> >
> > 2016-08-04 20:40:58 TaskConfig [INFO] No task.name.grouper.factory
> > configuration, using 'org.apache.samza.container.grouper.task.
> > GroupByContainerCountFactory'
> >
> > 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> > configuration from coordinator stream.
> >
> > 2016-08-04 20:40:58 JobCoordinator$ [INFO] Saving task-to-changelog
> > partition mapping: Map(Partition 0 -> 0)
> >
> > 2016-08-04 20:40:58 KafkaSystemProducer [INFO] Creating a new producer
> for
> > system kafka.
> >
> > 2016-08-04 20:40:58 ProducerConfig [INFO] ProducerConfig values:
> >
> > compression.type = none
> >
> > metric.reporters = []
> >
> > metadata.max.age.ms = 300000
> >
> > metadata.fetch.timeout.ms = 60000
> >
> > acks = 1
> >
> > batch.size = 16384
> >
> > reconnect.backoff.ms = 10
> >
> > bootstrap.servers = [sprdargas102:6667, sprdargas103:6667]
> >
> > receive.buffer.bytes = 32768
> >
> > retry.backoff.ms = 100
> >
> > buffer.memory = 33554432
> >
> > timeout.ms = 30000
> >
> > key.serializer = class org.apache.kafka.common.serialization.
> > ByteArraySerializer
> >
> > retries = 2147483647
> >
> > max.request.size = 1048576
> >
> > block.on.buffer.full = true
> >
> > value.serializer = class org.apache.kafka.common.serialization.
> > ByteArraySerializer
> >
> > metrics.sample.window.ms = 30000
> >
> > send.buffer.bytes = 131072
> >
> > max.in.flight.requests.per.connection = 1
> >
> > metrics.num.samples = 2
> >
> > linger.ms = 0
> >
> > client.id = samza_producer-Argos-1-1470343257285-4
> >
> >
> > 2016-08-04 20:40:58 ProducerConfig [WARN] The configuration
> > batch.num.messages = null was supplied but isn't a known config.
> >
> > 2016-08-04 20:40:58 ProducerConfig [WARN] The configuration producer.type
> > = null was supplied but isn't a known config.
> >
> > 2016-08-04 20:40:58 ProducerConfig [WARN] The configuration
> > metadata.broker.list = null was supplied but isn't a known config.
> >
> > 2016-08-04 20:40:59 TaskConfig [INFO] No task.name.grouper.factory
> > configuration, using 'org.apache.samza.container.grouper.task.
> > GroupByContainerCountFactory'
> >
> > 2016-08-04 20:40:59 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> > configuration from coordinator stream.
> >
> > 2016-08-04 20:41:00 JobCoordinator$ [INFO] Got change log stream
> metadata:
> > Map()
> >
> > 2016-08-04 20:41:00 SamzaAppMaster$ [INFO] got config:
> > {systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,
> > sprdargas103:2181,sprdargas104:2181, systems.kafka.samza.factory=
> > org.apache.samza.system.kafka.KafkaSystemFactory,
> > job.coordinator.system=kafka, systems.kafka.streams.metrics.
> samza.msg.serde=metrics,
> > job.id=1, systems.kafka.producer.producer.type=sync,
> > systems.kafka.producer.metadata.broker.list=sprdargas102:6667,
> sprdargas103:6667,
> > systems.kafka.producer.batch.num.messages=1,
> > systems.kafka.samza.msg.serde=json, systems.kafka.consumer.auto.
> offset.reset=largest,
> > systems.kafka.streams.PageViewEvent.samza.msg.serde=json,
> > systems.kafka.streams.ArgosRaw.samza.msg.serde=json, job.name=Argos,
> > systems.kafka.producer.bootstrap.servers=sprdargas102:6667,
> > sprdargas103:6667}
> >
> > 2016-08-04 20:41:00 ClientHelper [INFO] trying to connect to RM
> > 0.0.0.0:8032
> >
> > 2016-08-04 20:41:00 JmxServer [INFO] According to
> > Util.getLocalHost.getHostName we are sprdargas102
> >
> > - Shekar
> >
> > On Thu, Aug 4, 2016 at 2:23 PM, Yi Pan <ni...@gmail.com> wrote:
> >
> >> Hi, Shekar,
> >>
> >> Please see my replies below:
> >>
> >> On Thu, Aug 4, 2016 at 12:18 PM, Shekar Tippur <ct...@gmail.com>
> wrote:
> >>
> >> > Sorry .. got trigger happy.
> >> > Couple of observations:
> >> > 1. rmi port seem to be random ones Even with task.opts and setting the
> >> port
> >> > to 1099
> >> >
> >>
> >> You are right. Samza containers will choose a random port to start the
> >> JmxServer. We do have an issue w/ port conflicts in 0.10.0 and earlier
> >> versions. It is recently fixed in 0.10.1. The 0.10.1 RC is out for the
> >> vote
> >> now. It would be good if you can try it out. However, your exception
> stack
> >> trace indicates that it is not a port conflict, but a refusal of
> >> connection. Did you try telnet to localhost to see whether it can open
> the
> >> connection?
> >>
> >>
> >> > 2. Do I need to start rmiregistry manually?
> >> >
> >> No, you shouldn't need to start any other process manually for that.
> >>
> >> P.S. do you have a full log from your SamzaAppMaster? That would be
> >> helpful
> >> in debugging this issue.
> >>
> >>
> >> > 3. I dont have any firewall settings.
> >> >
> >> > On Thu, Aug 4, 2016 at 12:14 PM, Shekar Tippur <ct...@gmail.com>
> >> wrote:
> >> >
> >> > > Yi
> >> > > Thanks for responding.. I have tried diff options for task.opts and
> >> > > without it as well.
> >> > > I have tried this
> >> > >
> >> > > task.opts=-Dcom.sun.management.jmxremote -Dcom.sun.management.
> >> > jmxremote.port=1099
> >> > > -Dcom.sun.management.jmxremote.authenticate=false
> >> -Dcom.sun.management.
> >> > jmxremote.ssl=false
> >> > > -Djava.rmi.server.hostname=localhost
> >> > >
> >> > >
> >> > >
> >> > > Here is the property file:
> >> > > ##########################################################
> >> > >
> >> > > # Licensed to the Apache Software Foundation (ASF) under one
> >> > >
> >> > > # or more contributor license agreements. See the NOTICE file
> >> > >
> >> > > # distributed with this work for additional information
> >> > >
> >> > > # regarding copyright ownership. The ASF licenses this file
> >> > >
> >> > > # to you under the Apache License, Version 2.0 (the
> >> > >
> >> > > # "License"); you may not use this file except in compliance
> >> > >
> >> > > # with the License. You may obtain a copy of the License at
> >> > >
> >> > > #
> >> > >
> >> > > # http://www.apache.org/licenses/LICENSE-2.0
> >> > >
> >> > > #
> >> > >
> >> > > # Unless required by applicable law or agreed to in writing,
> >> > >
> >> > > # software distributed under the License is distributed on an
> >> > >
> >> > > # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> >> > >
> >> > > # KIND, either express or implied. See the License for the
> >> > >
> >> > > # specific language governing permissions and limitations
> >> > >
> >> > > # under the License.
> >> > >
> >> > >
> >> > > job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
> >> > >
> >> > > job.name=Argos
> >> > >
> >> > > #job.coordinator.system=kafka
> >> > >
> >> > > #job.coordinator.replication.factor=2
> >> > >
> >> > > # YARN
> >> > >
> >> > > yarn.package.path=http://localhost:8000/argos_aggregate.tar.gz
> >> > >
> >> > > #task.opts="-Dcom.sun.management.jmxremote -Dcom.sun.management.
> >> > jmxremote.port=7091
> >> > > -Dcom.sun.management.jmxremote.authenticate=false
> >> > > -Dcom.sun.management.jmxremote.ssl=false
> >> -Djava.rmi.server.hostname=loc
> >> > > alhost"
> >> > >
> >> > > # Task
> >> > >
> >> > > task.class=com.project.argos.task.ArgosParserStreamTask
> >> > >
> >> > > task.inputs=kafka.argos-raw
> >> > >
> >> > > task.outputs=argos-parser
> >> > >
> >> > > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.Ka
> >> > > fkaCheckpointManagerFactory
> >> > >
> >> > > task.checkpoint.system=kafka
> >> > >
> >> > > # Normally, this would be 3, but we have only one broker.
> >> > >
> >> > > task.checkpoint.replication.factor=2
> >> > >
> >> > > task.drop.deserialization.errors=true
> >> > >
> >> > > task.drop.serialization.errors=true
> >> > >
> >> > > task.ignored.exceptions=org.codehaus.jackson.JsonParseException
> >> > >
> >> > >
> >> > > # Samza10 properties
> >> > >
> >> > > #systems.kafka.producer.bootstrap.servers=localhost:6667
> >> > >
> >> > > systems.kafka.producer.bootstrap.servers=host1:6667,host1:6667
> >> > >
> >> > > job.coordinator.system=kafka
> >> > >
> >> > >
> >> > >
> >> > > # Metrics
> >> > >
> >> > > metrics.reporters=snapshot,jmx
> >> > >
> >> > > metrics.reporter.snapshot.class=org.apache.samza.metrics.
> >> > > reporter.MetricsSnapshotReporterFactory
> >> > >
> >> > > metrics.reporter.snapshot.stream=kafka.metrics
> >> > >
> >> > > metrics.reporter.jmx.class=org.apache.samza.metrics.reporter
> >> > > .JmxReporterFactory
> >> > >
> >> > >
> >> > > # Serializers
> >> > >
> >> > > serializers.registry.json.class=org.apache.samza.serializers
> >> > > .JsonSerdeFactory
> >> > >
> >> > > serializers.registry.string.class=org.apache.samza.serialize
> >> > > rs.StringSerdeFactory
> >> > >
> >> > > serializers.registry.metrics.class=org.apache.samza.serializ
> >> > > ers.MetricsSnapshotSerdeFactory
> >> > >
> >> > >
> >> > > # Systems
> >> > >
> >> > > systems.kafka.samza.factory=org.apache.samza.system.kafka.Ka
> >> > > fkaSystemFactory
> >> > >
> >> > > systems.kafka.samza.msg.serde=json
> >> > >
> >> > >
> >> > > systems.kafka.consumer.zookeeper.connect=host1:2181,
> >> > host2:2181,host3:2181
> >> > >
> >> > > systems.kafka.consumer.auto.offset.reset=largest
> >> > >
> >> > > systems.kafka.producer.producer.type=sync
> >> > >
> >> > > # Normally, we'd set this much higher, but we want things to look
> >> snappy
> >> > > in the demo.
> >> > >
> >> > > systems.kafka.producer.batch.num.messages=1
> >> > >
> >> > > systems.kafka.streams.metrics.samza.msg.serde=metrics
> >> > >
> >> > > systems.kafka.streams.ArgosRaw.samza.msg.serde=json
> >> > >
> >> > > systems.kafka.streams.PageViewEvent.samza.msg.serde=json
> >> > >
> >> > >
> >> > >
> >> > > # enable wildcard lookup on cache misses, will impact performance
> >> > >
> >> > > redis.wildcard.lookup=true
> >> > >
> >> > >
> >> > > # Property File
> >> > >
> >> > > load.property=true
> >> > >
> >> > >
> >> > >
> >> > > system.redis=host4
> >> > >
> >> > > systems.kafka.producer.metadata.broker.list=host1:6667host2:6667
> >> > >
> >> > > ##########################################################
> >> > >
> >> > > On Thu, Aug 4, 2016 at 11:19 AM, Yi Pan <ni...@gmail.com>
> wrote:
> >> > >
> >> > >> Hi, Shekar,
> >> > >>
> >> > >> Did you check your firewall configuration? Could you also paste
> your
> >> > >> configuration, especially task.opts?
> >> > >>
> >> > >> -Yi
> >> > >>
> >> > >> On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ct...@gmail.com>
> >> > wrote:
> >> > >>
> >> > >> > I am trying to submit a Samza job to yarn and I get a error:
> >> > >> >
> >> > >> > Exception in thread "main" java.io.IOException: Cannot bind to
> URL
> >> > >> > [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailabl
> >> > >> eException
> >> > >> > [Root exception is java.rmi.ConnectException: Connection refused
> to
> >> > >> host:
> >> > >> > localhost; nested exception is:
> >> > >> >
> >> > >> > java.net.ConnectException: Connection refused]
> >> > >> >
> >> > >> > at
> >> > >> > javax.management.remote.rmi.RMIConnectorServer.newIOException(
> >> > >> > RMIConnectorServer.java:827)
> >> > >> >
> >> > >> > at
> >> > >> > javax.management.remote.rmi.RMIConnectorServer.start(
> >> > >> > RMIConnectorServer.java:432)
> >> > >> >
> >> > >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
> >> > scala:89
> >> > >> )
> >> > >> >
> >> > >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
> >> > scala:43
> >> > >> )
> >> > >> >
> >> > >> > at
> >> > >> > org.apache.samza.job.yarn.SamzaAppMaster$.main(
> >> > SamzaAppMaster.scala:87)
> >> > >> >
> >> > >> > at
> >> > >> > org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster
> >> .scala)
> >> > >> >
> >> > >> > Caused by: javax.naming.ServiceUnavailableException [Root
> >> exception
> >> > is
> >> > >> > java.rmi.ConnectException: Connection refused to host: localhost;
> >> > nested
> >> > >> > exception is:
> >> > >> >
> >> > >> > java.net.ConnectException: Connection refused]
> >> > >> >
> >> > >> > at
> >> > >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
> >> > >> xt.java:147)
> >> > >> >
> >> > >> > at
> >> > >> > com.sun.jndi.toolkit.url.GenericURLContext.bind(
> >> > >> > GenericURLContext.java:228)
> >> > >> >
> >> > >> > at javax.naming.InitialContext.bi
> >> nd(InitialContext.java:425)
> >> > >> >
> >> > >> > at
> >> > >> > javax.management.remote.rmi.RMIConnectorServer.bind(
> >> > >> > RMIConnectorServer.java:644)
> >> > >> >
> >> > >> > at
> >> > >> > javax.management.remote.rmi.RMIConnectorServer.start(
> >> > >> > RMIConnectorServer.java:427)
> >> > >> >
> >> > >> > ... 4 more
> >> > >> >
> >> > >> > Caused by: java.rmi.ConnectException: Connection refused to host:
> >> > >> > localhost; nested exception is:
> >> > >> >
> >> > >> > java.net.ConnectException: Connection refused
> >> > >> >
> >> > >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> >> > >> > TCPEndpoint.java:619)
> >> > >> >
> >> > >> > at
> >> > >> > sun.rmi.transport.tcp.TCPChannel.createConnection(
> >> > TCPChannel.java:216)
> >> > >> >
> >> > >> > at
> >> > >> > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.
> >> java:202)
> >> > >> >
> >> > >> > at sun.rmi.server.UnicastRef.
> newCall(UnicastRef.java:342)
> >> > >> >
> >> > >> > at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown
> Source)
> >> > >> >
> >> > >> > at
> >> > >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
> >> > >> xt.java:141)
> >> > >> >
> >> > >> > ... 8 more
> >> > >> >
> >> > >> > Caused by: java.net.ConnectException: Connection refused
> >> > >> >
> >> > >> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> >> > >> >
> >> > >> > at
> >> > >> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
> >> > >> etImpl.java:
> >> > >> > 350)
> >> > >> >
> >> > >> > at
> >> > >> > java.net.AbstractPlainSocketImpl.connectToAddress(
> >> > >> > AbstractPlainSocketImpl.java:206)
> >> > >> >
> >> > >> > at
> >> > >> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
> >> > >> Impl.java:188)
> >> > >> >
> >> > >> > at java.net.SocksSocketImpl.conne
> >> ct(SocksSocketImpl.java:392)
> >> > >> >
> >> > >> > at java.net.Socket.connect(Socket.java:589)
> >> > >> >
> >> > >> > at java.net.Socket.connect(Socket.java:538)
> >> > >> >
> >> > >> > at java.net.Socket.<init>(Socket.java:434)
> >> > >> >
> >> > >> > at java.net.Socket.<init>(Socket.java:211)
> >> > >> >
> >> > >> > at
> >> > >> > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(
> >> > >> > RMIDirectSocketFactory.java:40)
> >> > >> >
> >> > >> > at
> >> > >> > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(
> >> > >> > RMIMasterSocketFactory.java:148)
> >> > >> >
> >> > >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> >> > >> > TCPEndpoint.java:613)
> >> > >> >
> >> > >> > ... 13 more
> >> > >> >
> >> > >> > Any idea on what could be the issue?
> >> > >> >
> >> > >> > - Shekar
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
Is there anyway we can disable this?
- Shekar
On Thu, Aug 4, 2016 at 4:02 PM, Shekar Tippur <ct...@gmail.com> wrote:
> Yi,
>
> I can do a curl or telnet to localhost but it is hard to simulate as it
> spawns random ports.
> The job fails.
>
> Here is the application log:
>
> 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got container id:
> container_1470270235966_0016_02_000001
>
> 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got app attempt id:
> appattempt_1470270235966_0016_000002
>
> 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager host:
> sprdargas102
>
> 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager port: 40531
>
> 2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager http port: 8042
>
> 2016-08-04 20:40:57 SamzaAppMaster$ [INFO] got coordinator system config:
> {systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,
> sprdargas103:2181,sprdargas104:2181, systems.kafka.samza.factory=
> org.apache.samza.system.kafka.KafkaSystemFactory,
> job.coordinator.system=kafka, systems.kafka.streams.metrics.samza.msg.serde=metrics,
> job.id=1, systems.kafka.producer.producer.type=sync,
> systems.kafka.producer.metadata.broker.list=sprdargas102:6667,sprdargas103:6667,
> systems.kafka.producer.batch.num.messages=1,
> systems.kafka.samza.msg.serde=json, systems.kafka.consumer.auto.offset.reset=largest,
> systems.kafka.streams.PageViewEvent.samza.msg.serde=json,
> systems.kafka.streams.ArgosRaw.samza.msg.serde=json, job.name=Argos,
> systems.kafka.producer.bootstrap.servers=sprdargas102:6667,
> sprdargas103:6667}
>
> 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> configuration contains 'metadata.broker.list'. This configuration is
> deprecated . Samza has been upgraded to use Kafka's new producer API.
> Please update your configurations based on the documentation at
> http://kafka.apache.org/documentation.html#newproducerconfigs
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
> is overridden to largest
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> overridden to samza_admin-Argos-1-1470343257174-0
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> overridden to undefined-samza-consumer-group-427db503-329d-4201-87da-
> eed83f9f1390
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
> is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
>
> 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> configuration contains 'metadata.broker.list'. This configuration is
> deprecated . Samza has been upgraded to use Kafka's new producer API.
> Please update your configurations based on the documentation at
> http://kafka.apache.org/documentation.html#newproducerconfigs
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
> is overridden to largest
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> overridden to samza_consumer-Argos-1-1470343257242-1
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> overridden to undefined-samza-consumer-group-aa7250b3-ffa6-4caa-8952-
> 6aaaa8c275f1
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
> is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
>
> 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> configuration contains 'metadata.broker.list'. This configuration is
> deprecated . Samza has been upgraded to use Kafka's new producer API.
> Please update your configurations based on the documentation at
> http://kafka.apache.org/documentation.html#newproducerconfigs
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
> is overridden to largest
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> overridden to samza_admin-Argos-1-1470343257270-2
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> overridden to undefined-samza-consumer-group-00ed785c-1ee2-41cd-82ec-
> 78c7a5e4f404
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
> is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
>
> 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> configuration contains 'metadata.broker.list'. This configuration is
> deprecated . Samza has been upgraded to use Kafka's new producer API.
> Please update your configurations based on the documentation at
> http://kafka.apache.org/documentation.html#newproducerconfigs
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
> is overridden to largest
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> overridden to samza_admin-Argos-1-1470343257283-3
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
> overridden to undefined-samza-consumer-group-d75f6419-4985-45d1-aeed-
> 18351ca05be5
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
> is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
>
> 2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer
> configuration contains 'metadata.broker.list'. This configuration is
> deprecated . Samza has been upgraded to use Kafka's new producer API.
> Please update your configurations based on the documentation at
> http://kafka.apache.org/documentation.html#newproducerconfigs
>
> 2016-08-04 20:40:57 JobCoordinator$ [INFO] Registering coordinator system
> stream.
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
> overridden to samza_admin-Argos-1-1470343257174-0
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> metadata.broker.list is overridden to sprdargas102:6667,sprdargas103:6667
>
> 2016-08-04 20:40:57 VerifiableProperties [INFO] Property
> request.timeout.ms is overridden to 30000
>
> 2016-08-04 20:40:57 ClientUtils$ [INFO] Fetching metadata from broker
> id:0,host:sprdargas102,port:6667 with correlation id 0 for 1 topic(s)
> Set(__samza_coordinator_Argos_1)
>
> 2016-08-04 20:40:57 SyncProducer [INFO] Connected to sprdargas102:6667 for
> producing
>
> 2016-08-04 20:40:57 SyncProducer [INFO] Disconnecting from
> sprdargas102:6667
>
> 2016-08-04 20:40:57 KafkaSystemAdmin$ [INFO] Got metadata:
> Map(__samza_coordinator_Argos_1 -> SystemStreamMetadata
> [streamName=__samza_coordinator_Argos_1, partitionMetadata={Partition
> [partition=0]=SystemStreamPartitionMetadata [oldestOffset=0,
> newestOffset=1269, upcomingOffset=1270]}])
>
> 2016-08-04 20:40:57 CoordinatorStreamSystemConsumer [INFO] Starting
> coordinator stream system consumer.
>
> 2016-08-04 20:40:57 KafkaSystemConsumer [INFO] Refreshing brokers for:
> Map([__samza_coordinator_Argos_1,0] -> 0)
>
> 2016-08-04 20:40:57 BrokerProxy [INFO] Creating new SimpleConsumer for
> host sprdargas102:6667 for system kafka
>
> 2016-08-04 20:40:57 GetOffset [INFO] Validating offset 0 for topic and
> partition [__samza_coordinator_Argos_1,0]
>
> 2016-08-04 20:40:57 GetOffset [INFO] Able to successfully read from offset
> 0 for topic and partition [__samza_coordinator_Argos_1,0]. Using it to
> instantiate consumer.
>
> 2016-08-04 20:40:57 BrokerProxy [INFO] Starting BrokerProxy for
> sprdargas102:6667
>
> 2016-08-04 20:40:57 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> configuration from coordinator stream.
>
> 2016-08-04 20:40:58 JobCoordinator$ [INFO] Got config:
> {systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory,
> system.redis=sprdargas104, task.drop.deserialization.errors=true,
> serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory,
> systems.kafka.streams.metrics.samza.msg.serde=metrics,
> serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory,
> task.checkpoint.system=kafka, task.checkpoint.factory=org.
> apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory,
> task.ignored.exceptions=org.codehaus.jackson.JsonParseException,
> systems.kafka.producer.metadata.broker.list=sprdargas102:6667,sprdargas103:6667,
> systems.kafka.samza.msg.serde=json, metrics.reporters=snapshot,jmx,
> systems.kafka.consumer.auto.offset.reset=largest, job.name=Argos,
> metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.
> MetricsSnapshotReporterFactory, systems.kafka.producer.bootstrap.servers=
> sprdargas102:6667,sprdargas103:6667, task.checkpoint.replication.factor=2,
> systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,
> sprdargas103:2181,sprdargas104:2181, load.property=true,
> job.coordinator.system=kafka, task.inputs=kafka.argos-raw,
> systems.kafka.producer.producer.type=sync, task.outputs=argos-parser,
> yarn.package.path=http://sprdargas102:8000/argos_aggregate.tar.gz,
> job.factory.class=org.apache.samza.job.yarn.YarnJobFactory,
> task.class=com.intuit.argos.task.ArgosParserStreamTask,
> redis.wildcard.lookup=true, metrics.reporter.jmx.class=
> org.apache.samza.metrics.reporter.JmxReporterFactory,
> systems.kafka.producer.batch.num.messages=1, systems.kafka.streams.
> PageViewEvent.samza.msg.serde=json, task.drop.serialization.errors=true,
> systems.kafka.streams.ArgosRaw.samza.msg.serde=json,
> serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory,
> task.opts=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099
> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
> -Djava.rmi.server.hostname=127.0.0.1, metrics.reporter.snapshot.
> stream=kafka.metrics}
>
> 2016-08-04 20:40:58 KafkaProducerConfig [WARN] Kafka producer
> configuration contains 'metadata.broker.list'. This configuration is
> deprecated . Samza has been upgraded to use Kafka's new producer API.
> Please update your configurations based on the documentation at
> http://kafka.apache.org/documentation.html#newproducerconfigs
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Verifying properties
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Property auto.offset.reset
> is overridden to largest
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Property client.id is
> overridden to samza_admin-Argos-1-1470343258775-5
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Property group.id is
> overridden to undefined-samza-consumer-group-b18c1f75-fd2a-4eea-bb28-
> 98d5a85fc3c8
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Property zookeeper.connect
> is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
>
> 2016-08-04 20:40:58 JobConfig [INFO] job.coordinator.replication.factor
> was not found. Using task.checkpoint.replication.factor=2 for coordinator
> stream
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Verifying properties
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Property client.id is
> overridden to samza_admin-Argos-1-1470343258775-5
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Property
> metadata.broker.list is overridden to sprdargas102:6667,sprdargas103:6667
>
> 2016-08-04 20:40:58 VerifiableProperties [INFO] Property
> request.timeout.ms is overridden to 30000
>
> 2016-08-04 20:40:58 ClientUtils$ [INFO] Fetching metadata from broker
> id:1,host:sprdargas103,port:6667 with correlation id 0 for 1 topic(s)
> Set(argos-raw)
>
> 2016-08-04 20:40:58 SyncProducer [INFO] Connected to sprdargas103:6667 for
> producing
>
> 2016-08-04 20:40:58 SyncProducer [INFO] Disconnecting from
> sprdargas103:6667
>
> 2016-08-04 20:40:58 KafkaSystemAdmin$ [INFO] Got metadata: Map(argos-raw
> -> SystemStreamMetadata [streamName=argos-raw, partitionMetadata={Partition
> [partition=0]=SystemStreamPartitionMetadata [oldestOffset=0,
> newestOffset=null, upcomingOffset=0]}])
>
> 2016-08-04 20:40:58 JobCoordinator$ [INFO] SystemStreamPartitionGrouper
> org.apache.samza.container.grouper.stream.GroupByPartition@5674e1f2 has
> grouped the SystemStreamPartitions into the following taskNames:
>
> 2016-08-04 20:40:58 CoordinatorStreamSystemProducer [INFO] Starting
> coordinator stream producer.
>
> 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Coordinator
> stream consumer already started
>
> 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> configuration from coordinator stream.
>
> 2016-08-04 20:40:58 CoordinatorStreamSystemProducer [INFO] Coordinator
> stream producer already started
>
> 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Coordinator
> stream consumer already started
>
> 2016-08-04 20:40:58 TaskConfig [INFO] No task.name.grouper.factory
> configuration, using 'org.apache.samza.container.grouper.task.
> GroupByContainerCountFactory'
>
> 2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> configuration from coordinator stream.
>
> 2016-08-04 20:40:58 JobCoordinator$ [INFO] Saving task-to-changelog
> partition mapping: Map(Partition 0 -> 0)
>
> 2016-08-04 20:40:58 KafkaSystemProducer [INFO] Creating a new producer for
> system kafka.
>
> 2016-08-04 20:40:58 ProducerConfig [INFO] ProducerConfig values:
>
> compression.type = none
>
> metric.reporters = []
>
> metadata.max.age.ms = 300000
>
> metadata.fetch.timeout.ms = 60000
>
> acks = 1
>
> batch.size = 16384
>
> reconnect.backoff.ms = 10
>
> bootstrap.servers = [sprdargas102:6667, sprdargas103:6667]
>
> receive.buffer.bytes = 32768
>
> retry.backoff.ms = 100
>
> buffer.memory = 33554432
>
> timeout.ms = 30000
>
> key.serializer = class org.apache.kafka.common.serialization.
> ByteArraySerializer
>
> retries = 2147483647
>
> max.request.size = 1048576
>
> block.on.buffer.full = true
>
> value.serializer = class org.apache.kafka.common.serialization.
> ByteArraySerializer
>
> metrics.sample.window.ms = 30000
>
> send.buffer.bytes = 131072
>
> max.in.flight.requests.per.connection = 1
>
> metrics.num.samples = 2
>
> linger.ms = 0
>
> client.id = samza_producer-Argos-1-1470343257285-4
>
>
> 2016-08-04 20:40:58 ProducerConfig [WARN] The configuration
> batch.num.messages = null was supplied but isn't a known config.
>
> 2016-08-04 20:40:58 ProducerConfig [WARN] The configuration producer.type
> = null was supplied but isn't a known config.
>
> 2016-08-04 20:40:58 ProducerConfig [WARN] The configuration
> metadata.broker.list = null was supplied but isn't a known config.
>
> 2016-08-04 20:40:59 TaskConfig [INFO] No task.name.grouper.factory
> configuration, using 'org.apache.samza.container.grouper.task.
> GroupByContainerCountFactory'
>
> 2016-08-04 20:40:59 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
> configuration from coordinator stream.
>
> 2016-08-04 20:41:00 JobCoordinator$ [INFO] Got change log stream metadata:
> Map()
>
> 2016-08-04 20:41:00 SamzaAppMaster$ [INFO] got config:
> {systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,
> sprdargas103:2181,sprdargas104:2181, systems.kafka.samza.factory=
> org.apache.samza.system.kafka.KafkaSystemFactory,
> job.coordinator.system=kafka, systems.kafka.streams.metrics.samza.msg.serde=metrics,
> job.id=1, systems.kafka.producer.producer.type=sync,
> systems.kafka.producer.metadata.broker.list=sprdargas102:6667,sprdargas103:6667,
> systems.kafka.producer.batch.num.messages=1,
> systems.kafka.samza.msg.serde=json, systems.kafka.consumer.auto.offset.reset=largest,
> systems.kafka.streams.PageViewEvent.samza.msg.serde=json,
> systems.kafka.streams.ArgosRaw.samza.msg.serde=json, job.name=Argos,
> systems.kafka.producer.bootstrap.servers=sprdargas102:6667,
> sprdargas103:6667}
>
> 2016-08-04 20:41:00 ClientHelper [INFO] trying to connect to RM
> 0.0.0.0:8032
>
> 2016-08-04 20:41:00 JmxServer [INFO] According to
> Util.getLocalHost.getHostName we are sprdargas102
>
> - Shekar
>
> On Thu, Aug 4, 2016 at 2:23 PM, Yi Pan <ni...@gmail.com> wrote:
>
>> Hi, Shekar,
>>
>> Please see my replies below:
>>
>> On Thu, Aug 4, 2016 at 12:18 PM, Shekar Tippur <ct...@gmail.com> wrote:
>>
>> > Sorry .. got trigger happy.
>> > Couple of observations:
>> > 1. rmi port seem to be random ones Even with task.opts and setting the
>> port
>> > to 1099
>> >
>>
>> You are right. Samza containers will choose a random port to start the
>> JmxServer. We do have an issue w/ port conflicts in 0.10.0 and earlier
>> versions. It is recently fixed in 0.10.1. The 0.10.1 RC is out for the
>> vote
>> now. It would be good if you can try it out. However, your exception stack
>> trace indicates that it is not a port conflict, but a refusal of
>> connection. Did you try telnet to localhost to see whether it can open the
>> connection?
>>
>>
>> > 2. Do I need to start rmiregistry manually?
>> >
>> No, you shouldn't need to start any other process manually for that.
>>
>> P.S. do you have a full log from your SamzaAppMaster? That would be
>> helpful
>> in debugging this issue.
>>
>>
>> > 3. I dont have any firewall settings.
>> >
>> > On Thu, Aug 4, 2016 at 12:14 PM, Shekar Tippur <ct...@gmail.com>
>> wrote:
>> >
>> > > Yi
>> > > Thanks for responding.. I have tried diff options for task.opts and
>> > > without it as well.
>> > > I have tried this
>> > >
>> > > task.opts=-Dcom.sun.management.jmxremote -Dcom.sun.management.
>> > jmxremote.port=1099
>> > > -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.
>> > jmxremote.ssl=false
>> > > -Djava.rmi.server.hostname=localhost
>> > >
>> > >
>> > >
>> > > Here is the property file:
>> > > ##########################################################
>> > >
>> > > # Licensed to the Apache Software Foundation (ASF) under one
>> > >
>> > > # or more contributor license agreements. See the NOTICE file
>> > >
>> > > # distributed with this work for additional information
>> > >
>> > > # regarding copyright ownership. The ASF licenses this file
>> > >
>> > > # to you under the Apache License, Version 2.0 (the
>> > >
>> > > # "License"); you may not use this file except in compliance
>> > >
>> > > # with the License. You may obtain a copy of the License at
>> > >
>> > > #
>> > >
>> > > # http://www.apache.org/licenses/LICENSE-2.0
>> > >
>> > > #
>> > >
>> > > # Unless required by applicable law or agreed to in writing,
>> > >
>> > > # software distributed under the License is distributed on an
>> > >
>> > > # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>> > >
>> > > # KIND, either express or implied. See the License for the
>> > >
>> > > # specific language governing permissions and limitations
>> > >
>> > > # under the License.
>> > >
>> > >
>> > > job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
>> > >
>> > > job.name=Argos
>> > >
>> > > #job.coordinator.system=kafka
>> > >
>> > > #job.coordinator.replication.factor=2
>> > >
>> > > # YARN
>> > >
>> > > yarn.package.path=http://localhost:8000/argos_aggregate.tar.gz
>> > >
>> > > #task.opts="-Dcom.sun.management.jmxremote -Dcom.sun.management.
>> > jmxremote.port=7091
>> > > -Dcom.sun.management.jmxremote.authenticate=false
>> > > -Dcom.sun.management.jmxremote.ssl=false
>> -Djava.rmi.server.hostname=loc
>> > > alhost"
>> > >
>> > > # Task
>> > >
>> > > task.class=com.project.argos.task.ArgosParserStreamTask
>> > >
>> > > task.inputs=kafka.argos-raw
>> > >
>> > > task.outputs=argos-parser
>> > >
>> > > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.Ka
>> > > fkaCheckpointManagerFactory
>> > >
>> > > task.checkpoint.system=kafka
>> > >
>> > > # Normally, this would be 3, but we have only one broker.
>> > >
>> > > task.checkpoint.replication.factor=2
>> > >
>> > > task.drop.deserialization.errors=true
>> > >
>> > > task.drop.serialization.errors=true
>> > >
>> > > task.ignored.exceptions=org.codehaus.jackson.JsonParseException
>> > >
>> > >
>> > > # Samza10 properties
>> > >
>> > > #systems.kafka.producer.bootstrap.servers=localhost:6667
>> > >
>> > > systems.kafka.producer.bootstrap.servers=host1:6667,host1:6667
>> > >
>> > > job.coordinator.system=kafka
>> > >
>> > >
>> > >
>> > > # Metrics
>> > >
>> > > metrics.reporters=snapshot,jmx
>> > >
>> > > metrics.reporter.snapshot.class=org.apache.samza.metrics.
>> > > reporter.MetricsSnapshotReporterFactory
>> > >
>> > > metrics.reporter.snapshot.stream=kafka.metrics
>> > >
>> > > metrics.reporter.jmx.class=org.apache.samza.metrics.reporter
>> > > .JmxReporterFactory
>> > >
>> > >
>> > > # Serializers
>> > >
>> > > serializers.registry.json.class=org.apache.samza.serializers
>> > > .JsonSerdeFactory
>> > >
>> > > serializers.registry.string.class=org.apache.samza.serialize
>> > > rs.StringSerdeFactory
>> > >
>> > > serializers.registry.metrics.class=org.apache.samza.serializ
>> > > ers.MetricsSnapshotSerdeFactory
>> > >
>> > >
>> > > # Systems
>> > >
>> > > systems.kafka.samza.factory=org.apache.samza.system.kafka.Ka
>> > > fkaSystemFactory
>> > >
>> > > systems.kafka.samza.msg.serde=json
>> > >
>> > >
>> > > systems.kafka.consumer.zookeeper.connect=host1:2181,
>> > host2:2181,host3:2181
>> > >
>> > > systems.kafka.consumer.auto.offset.reset=largest
>> > >
>> > > systems.kafka.producer.producer.type=sync
>> > >
>> > > # Normally, we'd set this much higher, but we want things to look
>> snappy
>> > > in the demo.
>> > >
>> > > systems.kafka.producer.batch.num.messages=1
>> > >
>> > > systems.kafka.streams.metrics.samza.msg.serde=metrics
>> > >
>> > > systems.kafka.streams.ArgosRaw.samza.msg.serde=json
>> > >
>> > > systems.kafka.streams.PageViewEvent.samza.msg.serde=json
>> > >
>> > >
>> > >
>> > > # enable wildcard lookup on cache misses, will impact performance
>> > >
>> > > redis.wildcard.lookup=true
>> > >
>> > >
>> > > # Property File
>> > >
>> > > load.property=true
>> > >
>> > >
>> > >
>> > > system.redis=host4
>> > >
>> > > systems.kafka.producer.metadata.broker.list=host1:6667host2:6667
>> > >
>> > > ##########################################################
>> > >
>> > > On Thu, Aug 4, 2016 at 11:19 AM, Yi Pan <ni...@gmail.com> wrote:
>> > >
>> > >> Hi, Shekar,
>> > >>
>> > >> Did you check your firewall configuration? Could you also paste your
>> > >> configuration, especially task.opts?
>> > >>
>> > >> -Yi
>> > >>
>> > >> On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ct...@gmail.com>
>> > wrote:
>> > >>
>> > >> > I am trying to submit a Samza job to yarn and I get a error:
>> > >> >
>> > >> > Exception in thread "main" java.io.IOException: Cannot bind to URL
>> > >> > [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailabl
>> > >> eException
>> > >> > [Root exception is java.rmi.ConnectException: Connection refused to
>> > >> host:
>> > >> > localhost; nested exception is:
>> > >> >
>> > >> > java.net.ConnectException: Connection refused]
>> > >> >
>> > >> > at
>> > >> > javax.management.remote.rmi.RMIConnectorServer.newIOException(
>> > >> > RMIConnectorServer.java:827)
>> > >> >
>> > >> > at
>> > >> > javax.management.remote.rmi.RMIConnectorServer.start(
>> > >> > RMIConnectorServer.java:432)
>> > >> >
>> > >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
>> > scala:89
>> > >> )
>> > >> >
>> > >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
>> > scala:43
>> > >> )
>> > >> >
>> > >> > at
>> > >> > org.apache.samza.job.yarn.SamzaAppMaster$.main(
>> > SamzaAppMaster.scala:87)
>> > >> >
>> > >> > at
>> > >> > org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster
>> .scala)
>> > >> >
>> > >> > Caused by: javax.naming.ServiceUnavailableException [Root
>> exception
>> > is
>> > >> > java.rmi.ConnectException: Connection refused to host: localhost;
>> > nested
>> > >> > exception is:
>> > >> >
>> > >> > java.net.ConnectException: Connection refused]
>> > >> >
>> > >> > at
>> > >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
>> > >> xt.java:147)
>> > >> >
>> > >> > at
>> > >> > com.sun.jndi.toolkit.url.GenericURLContext.bind(
>> > >> > GenericURLContext.java:228)
>> > >> >
>> > >> > at javax.naming.InitialContext.bi
>> nd(InitialContext.java:425)
>> > >> >
>> > >> > at
>> > >> > javax.management.remote.rmi.RMIConnectorServer.bind(
>> > >> > RMIConnectorServer.java:644)
>> > >> >
>> > >> > at
>> > >> > javax.management.remote.rmi.RMIConnectorServer.start(
>> > >> > RMIConnectorServer.java:427)
>> > >> >
>> > >> > ... 4 more
>> > >> >
>> > >> > Caused by: java.rmi.ConnectException: Connection refused to host:
>> > >> > localhost; nested exception is:
>> > >> >
>> > >> > java.net.ConnectException: Connection refused
>> > >> >
>> > >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
>> > >> > TCPEndpoint.java:619)
>> > >> >
>> > >> > at
>> > >> > sun.rmi.transport.tcp.TCPChannel.createConnection(
>> > TCPChannel.java:216)
>> > >> >
>> > >> > at
>> > >> > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.
>> java:202)
>> > >> >
>> > >> > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
>> > >> >
>> > >> > at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
>> > >> >
>> > >> > at
>> > >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
>> > >> xt.java:141)
>> > >> >
>> > >> > ... 8 more
>> > >> >
>> > >> > Caused by: java.net.ConnectException: Connection refused
>> > >> >
>> > >> > at java.net.PlainSocketImpl.socketConnect(Native Method)
>> > >> >
>> > >> > at
>> > >> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
>> > >> etImpl.java:
>> > >> > 350)
>> > >> >
>> > >> > at
>> > >> > java.net.AbstractPlainSocketImpl.connectToAddress(
>> > >> > AbstractPlainSocketImpl.java:206)
>> > >> >
>> > >> > at
>> > >> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
>> > >> Impl.java:188)
>> > >> >
>> > >> > at java.net.SocksSocketImpl.conne
>> ct(SocksSocketImpl.java:392)
>> > >> >
>> > >> > at java.net.Socket.connect(Socket.java:589)
>> > >> >
>> > >> > at java.net.Socket.connect(Socket.java:538)
>> > >> >
>> > >> > at java.net.Socket.<init>(Socket.java:434)
>> > >> >
>> > >> > at java.net.Socket.<init>(Socket.java:211)
>> > >> >
>> > >> > at
>> > >> > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(
>> > >> > RMIDirectSocketFactory.java:40)
>> > >> >
>> > >> > at
>> > >> > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(
>> > >> > RMIMasterSocketFactory.java:148)
>> > >> >
>> > >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
>> > >> > TCPEndpoint.java:613)
>> > >> >
>> > >> > ... 13 more
>> > >> >
>> > >> > Any idea on what could be the issue?
>> > >> >
>> > >> > - Shekar
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
Yi,
I can do a curl or telnet to localhost but it is hard to simulate as it
spawns random ports.
The job fails.
Here is the application log:
2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got container id:
container_1470270235966_0016_02_000001
2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got app attempt id:
appattempt_1470270235966_0016_000002
2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager host:
sprdargas102
2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager port: 40531
2016-08-04 20:40:56 SamzaAppMaster$ [INFO] got node manager http port: 8042
2016-08-04 20:40:57 SamzaAppMaster$ [INFO] got coordinator system config:
{systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,sprdargas103:2181,sprdargas104:2181,
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory,
job.coordinator.system=kafka,
systems.kafka.streams.metrics.samza.msg.serde=metrics, job.id=1,
systems.kafka.producer.producer.type=sync,
systems.kafka.producer.metadata.broker.list=sprdargas102:6667,sprdargas103:6667,
systems.kafka.producer.batch.num.messages=1,
systems.kafka.samza.msg.serde=json,
systems.kafka.consumer.auto.offset.reset=largest,
systems.kafka.streams.PageViewEvent.samza.msg.serde=json,
systems.kafka.streams.ArgosRaw.samza.msg.serde=json, job.name=Argos,
systems.kafka.producer.bootstrap.servers=sprdargas102:6667,sprdargas103:6667}
2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer configuration
contains 'metadata.broker.list'. This configuration is deprecated . Samza
has been upgraded to use Kafka's new producer API. Please update your
configurations based on the documentation at
http://kafka.apache.org/documentation.html#newproducerconfigs
2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
is overridden to largest
2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-Argos-1-1470343257174-0
2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
overridden to
undefined-samza-consumer-group-427db503-329d-4201-87da-eed83f9f1390
2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer configuration
contains 'metadata.broker.list'. This configuration is deprecated . Samza
has been upgraded to use Kafka's new producer API. Please update your
configurations based on the documentation at
http://kafka.apache.org/documentation.html#newproducerconfigs
2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
is overridden to largest
2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
overridden to samza_consumer-Argos-1-1470343257242-1
2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
overridden to
undefined-samza-consumer-group-aa7250b3-ffa6-4caa-8952-6aaaa8c275f1
2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer configuration
contains 'metadata.broker.list'. This configuration is deprecated . Samza
has been upgraded to use Kafka's new producer API. Please update your
configurations based on the documentation at
http://kafka.apache.org/documentation.html#newproducerconfigs
2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
is overridden to largest
2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-Argos-1-1470343257270-2
2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
overridden to
undefined-samza-consumer-group-00ed785c-1ee2-41cd-82ec-78c7a5e4f404
2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer configuration
contains 'metadata.broker.list'. This configuration is deprecated . Samza
has been upgraded to use Kafka's new producer API. Please update your
configurations based on the documentation at
http://kafka.apache.org/documentation.html#newproducerconfigs
2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
2016-08-04 20:40:57 VerifiableProperties [INFO] Property auto.offset.reset
is overridden to largest
2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-Argos-1-1470343257283-3
2016-08-04 20:40:57 VerifiableProperties [INFO] Property group.id is
overridden to
undefined-samza-consumer-group-d75f6419-4985-45d1-aeed-18351ca05be5
2016-08-04 20:40:57 VerifiableProperties [INFO] Property zookeeper.connect
is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
2016-08-04 20:40:57 KafkaProducerConfig [WARN] Kafka producer configuration
contains 'metadata.broker.list'. This configuration is deprecated . Samza
has been upgraded to use Kafka's new producer API. Please update your
configurations based on the documentation at
http://kafka.apache.org/documentation.html#newproducerconfigs
2016-08-04 20:40:57 JobCoordinator$ [INFO] Registering coordinator system
stream.
2016-08-04 20:40:57 VerifiableProperties [INFO] Verifying properties
2016-08-04 20:40:57 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-Argos-1-1470343257174-0
2016-08-04 20:40:57 VerifiableProperties [INFO] Property
metadata.broker.list is overridden to sprdargas102:6667,sprdargas103:6667
2016-08-04 20:40:57 VerifiableProperties [INFO] Property request.timeout.ms
is overridden to 30000
2016-08-04 20:40:57 ClientUtils$ [INFO] Fetching metadata from broker
id:0,host:sprdargas102,port:6667 with correlation id 0 for 1 topic(s)
Set(__samza_coordinator_Argos_1)
2016-08-04 20:40:57 SyncProducer [INFO] Connected to sprdargas102:6667 for
producing
2016-08-04 20:40:57 SyncProducer [INFO] Disconnecting from sprdargas102:6667
2016-08-04 20:40:57 KafkaSystemAdmin$ [INFO] Got metadata:
Map(__samza_coordinator_Argos_1 -> SystemStreamMetadata
[streamName=__samza_coordinator_Argos_1, partitionMetadata={Partition
[partition=0]=SystemStreamPartitionMetadata [oldestOffset=0,
newestOffset=1269, upcomingOffset=1270]}])
2016-08-04 20:40:57 CoordinatorStreamSystemConsumer [INFO] Starting
coordinator stream system consumer.
2016-08-04 20:40:57 KafkaSystemConsumer [INFO] Refreshing brokers for:
Map([__samza_coordinator_Argos_1,0] -> 0)
2016-08-04 20:40:57 BrokerProxy [INFO] Creating new SimpleConsumer for host
sprdargas102:6667 for system kafka
2016-08-04 20:40:57 GetOffset [INFO] Validating offset 0 for topic and
partition [__samza_coordinator_Argos_1,0]
2016-08-04 20:40:57 GetOffset [INFO] Able to successfully read from offset
0 for topic and partition [__samza_coordinator_Argos_1,0]. Using it to
instantiate consumer.
2016-08-04 20:40:57 BrokerProxy [INFO] Starting BrokerProxy for
sprdargas102:6667
2016-08-04 20:40:57 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
configuration from coordinator stream.
2016-08-04 20:40:58 JobCoordinator$ [INFO] Got config:
{systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory,
system.redis=sprdargas104, task.drop.deserialization.errors=true,
serializers.registry.metrics.class=org.apache.samza.serializers.MetricsSnapshotSerdeFactory,
systems.kafka.streams.metrics.samza.msg.serde=metrics,
serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory,
task.checkpoint.system=kafka,
task.checkpoint.factory=org.apache.samza.checkpoint.kafka.KafkaCheckpointManagerFactory,
task.ignored.exceptions=org.codehaus.jackson.JsonParseException,
systems.kafka.producer.metadata.broker.list=sprdargas102:6667,sprdargas103:6667,
systems.kafka.samza.msg.serde=json, metrics.reporters=snapshot,jmx,
systems.kafka.consumer.auto.offset.reset=largest, job.name=Argos,
metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory,
systems.kafka.producer.bootstrap.servers=sprdargas102:6667,sprdargas103:6667,
task.checkpoint.replication.factor=2,
systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,sprdargas103:2181,sprdargas104:2181,
load.property=true, job.coordinator.system=kafka,
task.inputs=kafka.argos-raw, systems.kafka.producer.producer.type=sync,
task.outputs=argos-parser, yarn.package.path=
http://sprdargas102:8000/argos_aggregate.tar.gz,
job.factory.class=org.apache.samza.job.yarn.YarnJobFactory,
task.class=com.intuit.argos.task.ArgosParserStreamTask,
redis.wildcard.lookup=true,
metrics.reporter.jmx.class=org.apache.samza.metrics.reporter.JmxReporterFactory,
systems.kafka.producer.batch.num.messages=1,
systems.kafka.streams.PageViewEvent.samza.msg.serde=json,
task.drop.serialization.errors=true,
systems.kafka.streams.ArgosRaw.samza.msg.serde=json,
serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory,
task.opts=-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=127.0.0.1,
metrics.reporter.snapshot.stream=kafka.metrics}
2016-08-04 20:40:58 KafkaProducerConfig [WARN] Kafka producer configuration
contains 'metadata.broker.list'. This configuration is deprecated . Samza
has been upgraded to use Kafka's new producer API. Please update your
configurations based on the documentation at
http://kafka.apache.org/documentation.html#newproducerconfigs
2016-08-04 20:40:58 VerifiableProperties [INFO] Verifying properties
2016-08-04 20:40:58 VerifiableProperties [INFO] Property auto.offset.reset
is overridden to largest
2016-08-04 20:40:58 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-Argos-1-1470343258775-5
2016-08-04 20:40:58 VerifiableProperties [INFO] Property group.id is
overridden to
undefined-samza-consumer-group-b18c1f75-fd2a-4eea-bb28-98d5a85fc3c8
2016-08-04 20:40:58 VerifiableProperties [INFO] Property zookeeper.connect
is overridden to sprdargas102:2181,sprdargas103:2181,sprdargas104:2181
2016-08-04 20:40:58 JobConfig [INFO] job.coordinator.replication.factor was
not found. Using task.checkpoint.replication.factor=2 for coordinator stream
2016-08-04 20:40:58 VerifiableProperties [INFO] Verifying properties
2016-08-04 20:40:58 VerifiableProperties [INFO] Property client.id is
overridden to samza_admin-Argos-1-1470343258775-5
2016-08-04 20:40:58 VerifiableProperties [INFO] Property
metadata.broker.list is overridden to sprdargas102:6667,sprdargas103:6667
2016-08-04 20:40:58 VerifiableProperties [INFO] Property request.timeout.ms
is overridden to 30000
2016-08-04 20:40:58 ClientUtils$ [INFO] Fetching metadata from broker
id:1,host:sprdargas103,port:6667 with correlation id 0 for 1 topic(s)
Set(argos-raw)
2016-08-04 20:40:58 SyncProducer [INFO] Connected to sprdargas103:6667 for
producing
2016-08-04 20:40:58 SyncProducer [INFO] Disconnecting from sprdargas103:6667
2016-08-04 20:40:58 KafkaSystemAdmin$ [INFO] Got metadata: Map(argos-raw ->
SystemStreamMetadata [streamName=argos-raw, partitionMetadata={Partition
[partition=0]=SystemStreamPartitionMetadata [oldestOffset=0,
newestOffset=null, upcomingOffset=0]}])
2016-08-04 20:40:58 JobCoordinator$ [INFO] SystemStreamPartitionGrouper
org.apache.samza.container.grouper.stream.GroupByPartition@5674e1f2 has
grouped the SystemStreamPartitions into the following taskNames:
2016-08-04 20:40:58 CoordinatorStreamSystemProducer [INFO] Starting
coordinator stream producer.
2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Coordinator
stream consumer already started
2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
configuration from coordinator stream.
2016-08-04 20:40:58 CoordinatorStreamSystemProducer [INFO] Coordinator
stream producer already started
2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Coordinator
stream consumer already started
2016-08-04 20:40:58 TaskConfig [INFO] No task.name.grouper.factory
configuration, using
'org.apache.samza.container.grouper.task.GroupByContainerCountFactory'
2016-08-04 20:40:58 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
configuration from coordinator stream.
2016-08-04 20:40:58 JobCoordinator$ [INFO] Saving task-to-changelog
partition mapping: Map(Partition 0 -> 0)
2016-08-04 20:40:58 KafkaSystemProducer [INFO] Creating a new producer for
system kafka.
2016-08-04 20:40:58 ProducerConfig [INFO] ProducerConfig values:
compression.type = none
metric.reporters = []
metadata.max.age.ms = 300000
metadata.fetch.timeout.ms = 60000
acks = 1
batch.size = 16384
reconnect.backoff.ms = 10
bootstrap.servers = [sprdargas102:6667, sprdargas103:6667]
receive.buffer.bytes = 32768
retry.backoff.ms = 100
buffer.memory = 33554432
timeout.ms = 30000
key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
retries = 2147483647
max.request.size = 1048576
block.on.buffer.full = true
value.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
metrics.sample.window.ms = 30000
send.buffer.bytes = 131072
max.in.flight.requests.per.connection = 1
metrics.num.samples = 2
linger.ms = 0
client.id = samza_producer-Argos-1-1470343257285-4
2016-08-04 20:40:58 ProducerConfig [WARN] The configuration
batch.num.messages = null was supplied but isn't a known config.
2016-08-04 20:40:58 ProducerConfig [WARN] The configuration producer.type =
null was supplied but isn't a known config.
2016-08-04 20:40:58 ProducerConfig [WARN] The configuration
metadata.broker.list = null was supplied but isn't a known config.
2016-08-04 20:40:59 TaskConfig [INFO] No task.name.grouper.factory
configuration, using
'org.apache.samza.container.grouper.task.GroupByContainerCountFactory'
2016-08-04 20:40:59 CoordinatorStreamSystemConsumer [INFO] Bootstrapping
configuration from coordinator stream.
2016-08-04 20:41:00 JobCoordinator$ [INFO] Got change log stream metadata:
Map()
2016-08-04 20:41:00 SamzaAppMaster$ [INFO] got config:
{systems.kafka.consumer.zookeeper.connect=sprdargas102:2181,sprdargas103:2181,sprdargas104:2181,
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory,
job.coordinator.system=kafka,
systems.kafka.streams.metrics.samza.msg.serde=metrics, job.id=1,
systems.kafka.producer.producer.type=sync,
systems.kafka.producer.metadata.broker.list=sprdargas102:6667,sprdargas103:6667,
systems.kafka.producer.batch.num.messages=1,
systems.kafka.samza.msg.serde=json,
systems.kafka.consumer.auto.offset.reset=largest,
systems.kafka.streams.PageViewEvent.samza.msg.serde=json,
systems.kafka.streams.ArgosRaw.samza.msg.serde=json, job.name=Argos,
systems.kafka.producer.bootstrap.servers=sprdargas102:6667,sprdargas103:6667}
2016-08-04 20:41:00 ClientHelper [INFO] trying to connect to RM 0.0.0.0:8032
2016-08-04 20:41:00 JmxServer [INFO] According to
Util.getLocalHost.getHostName we are sprdargas102
- Shekar
On Thu, Aug 4, 2016 at 2:23 PM, Yi Pan <ni...@gmail.com> wrote:
> Hi, Shekar,
>
> Please see my replies below:
>
> On Thu, Aug 4, 2016 at 12:18 PM, Shekar Tippur <ct...@gmail.com> wrote:
>
> > Sorry .. got trigger happy.
> > Couple of observations:
> > 1. rmi port seem to be random ones Even with task.opts and setting the
> port
> > to 1099
> >
>
> You are right. Samza containers will choose a random port to start the
> JmxServer. We do have an issue w/ port conflicts in 0.10.0 and earlier
> versions. It is recently fixed in 0.10.1. The 0.10.1 RC is out for the vote
> now. It would be good if you can try it out. However, your exception stack
> trace indicates that it is not a port conflict, but a refusal of
> connection. Did you try telnet to localhost to see whether it can open the
> connection?
>
>
> > 2. Do I need to start rmiregistry manually?
> >
> No, you shouldn't need to start any other process manually for that.
>
> P.S. do you have a full log from your SamzaAppMaster? That would be helpful
> in debugging this issue.
>
>
> > 3. I dont have any firewall settings.
> >
> > On Thu, Aug 4, 2016 at 12:14 PM, Shekar Tippur <ct...@gmail.com>
> wrote:
> >
> > > Yi
> > > Thanks for responding.. I have tried diff options for task.opts and
> > > without it as well.
> > > I have tried this
> > >
> > > task.opts=-Dcom.sun.management.jmxremote -Dcom.sun.management.
> > jmxremote.port=1099
> > > -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.
> > jmxremote.ssl=false
> > > -Djava.rmi.server.hostname=localhost
> > >
> > >
> > >
> > > Here is the property file:
> > > ##########################################################
> > >
> > > # Licensed to the Apache Software Foundation (ASF) under one
> > >
> > > # or more contributor license agreements. See the NOTICE file
> > >
> > > # distributed with this work for additional information
> > >
> > > # regarding copyright ownership. The ASF licenses this file
> > >
> > > # to you under the Apache License, Version 2.0 (the
> > >
> > > # "License"); you may not use this file except in compliance
> > >
> > > # with the License. You may obtain a copy of the License at
> > >
> > > #
> > >
> > > # http://www.apache.org/licenses/LICENSE-2.0
> > >
> > > #
> > >
> > > # Unless required by applicable law or agreed to in writing,
> > >
> > > # software distributed under the License is distributed on an
> > >
> > > # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> > >
> > > # KIND, either express or implied. See the License for the
> > >
> > > # specific language governing permissions and limitations
> > >
> > > # under the License.
> > >
> > >
> > > job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
> > >
> > > job.name=Argos
> > >
> > > #job.coordinator.system=kafka
> > >
> > > #job.coordinator.replication.factor=2
> > >
> > > # YARN
> > >
> > > yarn.package.path=http://localhost:8000/argos_aggregate.tar.gz
> > >
> > > #task.opts="-Dcom.sun.management.jmxremote -Dcom.sun.management.
> > jmxremote.port=7091
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > -Dcom.sun.management.jmxremote.ssl=false
> -Djava.rmi.server.hostname=loc
> > > alhost"
> > >
> > > # Task
> > >
> > > task.class=com.project.argos.task.ArgosParserStreamTask
> > >
> > > task.inputs=kafka.argos-raw
> > >
> > > task.outputs=argos-parser
> > >
> > > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.Ka
> > > fkaCheckpointManagerFactory
> > >
> > > task.checkpoint.system=kafka
> > >
> > > # Normally, this would be 3, but we have only one broker.
> > >
> > > task.checkpoint.replication.factor=2
> > >
> > > task.drop.deserialization.errors=true
> > >
> > > task.drop.serialization.errors=true
> > >
> > > task.ignored.exceptions=org.codehaus.jackson.JsonParseException
> > >
> > >
> > > # Samza10 properties
> > >
> > > #systems.kafka.producer.bootstrap.servers=localhost:6667
> > >
> > > systems.kafka.producer.bootstrap.servers=host1:6667,host1:6667
> > >
> > > job.coordinator.system=kafka
> > >
> > >
> > >
> > > # Metrics
> > >
> > > metrics.reporters=snapshot,jmx
> > >
> > > metrics.reporter.snapshot.class=org.apache.samza.metrics.
> > > reporter.MetricsSnapshotReporterFactory
> > >
> > > metrics.reporter.snapshot.stream=kafka.metrics
> > >
> > > metrics.reporter.jmx.class=org.apache.samza.metrics.reporter
> > > .JmxReporterFactory
> > >
> > >
> > > # Serializers
> > >
> > > serializers.registry.json.class=org.apache.samza.serializers
> > > .JsonSerdeFactory
> > >
> > > serializers.registry.string.class=org.apache.samza.serialize
> > > rs.StringSerdeFactory
> > >
> > > serializers.registry.metrics.class=org.apache.samza.serializ
> > > ers.MetricsSnapshotSerdeFactory
> > >
> > >
> > > # Systems
> > >
> > > systems.kafka.samza.factory=org.apache.samza.system.kafka.Ka
> > > fkaSystemFactory
> > >
> > > systems.kafka.samza.msg.serde=json
> > >
> > >
> > > systems.kafka.consumer.zookeeper.connect=host1:2181,
> > host2:2181,host3:2181
> > >
> > > systems.kafka.consumer.auto.offset.reset=largest
> > >
> > > systems.kafka.producer.producer.type=sync
> > >
> > > # Normally, we'd set this much higher, but we want things to look
> snappy
> > > in the demo.
> > >
> > > systems.kafka.producer.batch.num.messages=1
> > >
> > > systems.kafka.streams.metrics.samza.msg.serde=metrics
> > >
> > > systems.kafka.streams.ArgosRaw.samza.msg.serde=json
> > >
> > > systems.kafka.streams.PageViewEvent.samza.msg.serde=json
> > >
> > >
> > >
> > > # enable wildcard lookup on cache misses, will impact performance
> > >
> > > redis.wildcard.lookup=true
> > >
> > >
> > > # Property File
> > >
> > > load.property=true
> > >
> > >
> > >
> > > system.redis=host4
> > >
> > > systems.kafka.producer.metadata.broker.list=host1:6667host2:6667
> > >
> > > ##########################################################
> > >
> > > On Thu, Aug 4, 2016 at 11:19 AM, Yi Pan <ni...@gmail.com> wrote:
> > >
> > >> Hi, Shekar,
> > >>
> > >> Did you check your firewall configuration? Could you also paste your
> > >> configuration, especially task.opts?
> > >>
> > >> -Yi
> > >>
> > >> On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ct...@gmail.com>
> > wrote:
> > >>
> > >> > I am trying to submit a Samza job to yarn and I get a error:
> > >> >
> > >> > Exception in thread "main" java.io.IOException: Cannot bind to URL
> > >> > [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailabl
> > >> eException
> > >> > [Root exception is java.rmi.ConnectException: Connection refused to
> > >> host:
> > >> > localhost; nested exception is:
> > >> >
> > >> > java.net.ConnectException: Connection refused]
> > >> >
> > >> > at
> > >> > javax.management.remote.rmi.RMIConnectorServer.newIOException(
> > >> > RMIConnectorServer.java:827)
> > >> >
> > >> > at
> > >> > javax.management.remote.rmi.RMIConnectorServer.start(
> > >> > RMIConnectorServer.java:432)
> > >> >
> > >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
> > scala:89
> > >> )
> > >> >
> > >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
> > scala:43
> > >> )
> > >> >
> > >> > at
> > >> > org.apache.samza.job.yarn.SamzaAppMaster$.main(
> > SamzaAppMaster.scala:87)
> > >> >
> > >> > at
> > >> > org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
> > >> >
> > >> > Caused by: javax.naming.ServiceUnavailableException [Root exception
> > is
> > >> > java.rmi.ConnectException: Connection refused to host: localhost;
> > nested
> > >> > exception is:
> > >> >
> > >> > java.net.ConnectException: Connection refused]
> > >> >
> > >> > at
> > >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
> > >> xt.java:147)
> > >> >
> > >> > at
> > >> > com.sun.jndi.toolkit.url.GenericURLContext.bind(
> > >> > GenericURLContext.java:228)
> > >> >
> > >> > at javax.naming.InitialContext.
> bind(InitialContext.java:425)
> > >> >
> > >> > at
> > >> > javax.management.remote.rmi.RMIConnectorServer.bind(
> > >> > RMIConnectorServer.java:644)
> > >> >
> > >> > at
> > >> > javax.management.remote.rmi.RMIConnectorServer.start(
> > >> > RMIConnectorServer.java:427)
> > >> >
> > >> > ... 4 more
> > >> >
> > >> > Caused by: java.rmi.ConnectException: Connection refused to host:
> > >> > localhost; nested exception is:
> > >> >
> > >> > java.net.ConnectException: Connection refused
> > >> >
> > >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> > >> > TCPEndpoint.java:619)
> > >> >
> > >> > at
> > >> > sun.rmi.transport.tcp.TCPChannel.createConnection(
> > TCPChannel.java:216)
> > >> >
> > >> > at
> > >> > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> > >> >
> > >> > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
> > >> >
> > >> > at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
> > >> >
> > >> > at
> > >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
> > >> xt.java:141)
> > >> >
> > >> > ... 8 more
> > >> >
> > >> > Caused by: java.net.ConnectException: Connection refused
> > >> >
> > >> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> > >> >
> > >> > at
> > >> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
> > >> etImpl.java:
> > >> > 350)
> > >> >
> > >> > at
> > >> > java.net.AbstractPlainSocketImpl.connectToAddress(
> > >> > AbstractPlainSocketImpl.java:206)
> > >> >
> > >> > at
> > >> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
> > >> Impl.java:188)
> > >> >
> > >> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:
> 392)
> > >> >
> > >> > at java.net.Socket.connect(Socket.java:589)
> > >> >
> > >> > at java.net.Socket.connect(Socket.java:538)
> > >> >
> > >> > at java.net.Socket.<init>(Socket.java:434)
> > >> >
> > >> > at java.net.Socket.<init>(Socket.java:211)
> > >> >
> > >> > at
> > >> > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(
> > >> > RMIDirectSocketFactory.java:40)
> > >> >
> > >> > at
> > >> > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(
> > >> > RMIMasterSocketFactory.java:148)
> > >> >
> > >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> > >> > TCPEndpoint.java:613)
> > >> >
> > >> > ... 13 more
> > >> >
> > >> > Any idea on what could be the issue?
> > >> >
> > >> > - Shekar
> > >> >
> > >>
> > >
> > >
> >
>
Re: Samza yarn job - cannot bind to local host
Posted by Yi Pan <ni...@gmail.com>.
Hi, Shekar,
Please see my replies below:
On Thu, Aug 4, 2016 at 12:18 PM, Shekar Tippur <ct...@gmail.com> wrote:
> Sorry .. got trigger happy.
> Couple of observations:
> 1. rmi port seem to be random ones Even with task.opts and setting the port
> to 1099
>
You are right. Samza containers will choose a random port to start the
JmxServer. We do have an issue w/ port conflicts in 0.10.0 and earlier
versions. It is recently fixed in 0.10.1. The 0.10.1 RC is out for the vote
now. It would be good if you can try it out. However, your exception stack
trace indicates that it is not a port conflict, but a refusal of
connection. Did you try telnet to localhost to see whether it can open the
connection?
> 2. Do I need to start rmiregistry manually?
>
No, you shouldn't need to start any other process manually for that.
P.S. do you have a full log from your SamzaAppMaster? That would be helpful
in debugging this issue.
> 3. I dont have any firewall settings.
>
> On Thu, Aug 4, 2016 at 12:14 PM, Shekar Tippur <ct...@gmail.com> wrote:
>
> > Yi
> > Thanks for responding.. I have tried diff options for task.opts and
> > without it as well.
> > I have tried this
> >
> > task.opts=-Dcom.sun.management.jmxremote -Dcom.sun.management.
> jmxremote.port=1099
> > -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.
> jmxremote.ssl=false
> > -Djava.rmi.server.hostname=localhost
> >
> >
> >
> > Here is the property file:
> > ##########################################################
> >
> > # Licensed to the Apache Software Foundation (ASF) under one
> >
> > # or more contributor license agreements. See the NOTICE file
> >
> > # distributed with this work for additional information
> >
> > # regarding copyright ownership. The ASF licenses this file
> >
> > # to you under the Apache License, Version 2.0 (the
> >
> > # "License"); you may not use this file except in compliance
> >
> > # with the License. You may obtain a copy of the License at
> >
> > #
> >
> > # http://www.apache.org/licenses/LICENSE-2.0
> >
> > #
> >
> > # Unless required by applicable law or agreed to in writing,
> >
> > # software distributed under the License is distributed on an
> >
> > # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
> >
> > # KIND, either express or implied. See the License for the
> >
> > # specific language governing permissions and limitations
> >
> > # under the License.
> >
> >
> > job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
> >
> > job.name=Argos
> >
> > #job.coordinator.system=kafka
> >
> > #job.coordinator.replication.factor=2
> >
> > # YARN
> >
> > yarn.package.path=http://localhost:8000/argos_aggregate.tar.gz
> >
> > #task.opts="-Dcom.sun.management.jmxremote -Dcom.sun.management.
> jmxremote.port=7091
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=loc
> > alhost"
> >
> > # Task
> >
> > task.class=com.project.argos.task.ArgosParserStreamTask
> >
> > task.inputs=kafka.argos-raw
> >
> > task.outputs=argos-parser
> >
> > task.checkpoint.factory=org.apache.samza.checkpoint.kafka.Ka
> > fkaCheckpointManagerFactory
> >
> > task.checkpoint.system=kafka
> >
> > # Normally, this would be 3, but we have only one broker.
> >
> > task.checkpoint.replication.factor=2
> >
> > task.drop.deserialization.errors=true
> >
> > task.drop.serialization.errors=true
> >
> > task.ignored.exceptions=org.codehaus.jackson.JsonParseException
> >
> >
> > # Samza10 properties
> >
> > #systems.kafka.producer.bootstrap.servers=localhost:6667
> >
> > systems.kafka.producer.bootstrap.servers=host1:6667,host1:6667
> >
> > job.coordinator.system=kafka
> >
> >
> >
> > # Metrics
> >
> > metrics.reporters=snapshot,jmx
> >
> > metrics.reporter.snapshot.class=org.apache.samza.metrics.
> > reporter.MetricsSnapshotReporterFactory
> >
> > metrics.reporter.snapshot.stream=kafka.metrics
> >
> > metrics.reporter.jmx.class=org.apache.samza.metrics.reporter
> > .JmxReporterFactory
> >
> >
> > # Serializers
> >
> > serializers.registry.json.class=org.apache.samza.serializers
> > .JsonSerdeFactory
> >
> > serializers.registry.string.class=org.apache.samza.serialize
> > rs.StringSerdeFactory
> >
> > serializers.registry.metrics.class=org.apache.samza.serializ
> > ers.MetricsSnapshotSerdeFactory
> >
> >
> > # Systems
> >
> > systems.kafka.samza.factory=org.apache.samza.system.kafka.Ka
> > fkaSystemFactory
> >
> > systems.kafka.samza.msg.serde=json
> >
> >
> > systems.kafka.consumer.zookeeper.connect=host1:2181,
> host2:2181,host3:2181
> >
> > systems.kafka.consumer.auto.offset.reset=largest
> >
> > systems.kafka.producer.producer.type=sync
> >
> > # Normally, we'd set this much higher, but we want things to look snappy
> > in the demo.
> >
> > systems.kafka.producer.batch.num.messages=1
> >
> > systems.kafka.streams.metrics.samza.msg.serde=metrics
> >
> > systems.kafka.streams.ArgosRaw.samza.msg.serde=json
> >
> > systems.kafka.streams.PageViewEvent.samza.msg.serde=json
> >
> >
> >
> > # enable wildcard lookup on cache misses, will impact performance
> >
> > redis.wildcard.lookup=true
> >
> >
> > # Property File
> >
> > load.property=true
> >
> >
> >
> > system.redis=host4
> >
> > systems.kafka.producer.metadata.broker.list=host1:6667host2:6667
> >
> > ##########################################################
> >
> > On Thu, Aug 4, 2016 at 11:19 AM, Yi Pan <ni...@gmail.com> wrote:
> >
> >> Hi, Shekar,
> >>
> >> Did you check your firewall configuration? Could you also paste your
> >> configuration, especially task.opts?
> >>
> >> -Yi
> >>
> >> On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ct...@gmail.com>
> wrote:
> >>
> >> > I am trying to submit a Samza job to yarn and I get a error:
> >> >
> >> > Exception in thread "main" java.io.IOException: Cannot bind to URL
> >> > [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailabl
> >> eException
> >> > [Root exception is java.rmi.ConnectException: Connection refused to
> >> host:
> >> > localhost; nested exception is:
> >> >
> >> > java.net.ConnectException: Connection refused]
> >> >
> >> > at
> >> > javax.management.remote.rmi.RMIConnectorServer.newIOException(
> >> > RMIConnectorServer.java:827)
> >> >
> >> > at
> >> > javax.management.remote.rmi.RMIConnectorServer.start(
> >> > RMIConnectorServer.java:432)
> >> >
> >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
> scala:89
> >> )
> >> >
> >> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.
> scala:43
> >> )
> >> >
> >> > at
> >> > org.apache.samza.job.yarn.SamzaAppMaster$.main(
> SamzaAppMaster.scala:87)
> >> >
> >> > at
> >> > org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
> >> >
> >> > Caused by: javax.naming.ServiceUnavailableException [Root exception
> is
> >> > java.rmi.ConnectException: Connection refused to host: localhost;
> nested
> >> > exception is:
> >> >
> >> > java.net.ConnectException: Connection refused]
> >> >
> >> > at
> >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
> >> xt.java:147)
> >> >
> >> > at
> >> > com.sun.jndi.toolkit.url.GenericURLContext.bind(
> >> > GenericURLContext.java:228)
> >> >
> >> > at javax.naming.InitialContext.bind(InitialContext.java:425)
> >> >
> >> > at
> >> > javax.management.remote.rmi.RMIConnectorServer.bind(
> >> > RMIConnectorServer.java:644)
> >> >
> >> > at
> >> > javax.management.remote.rmi.RMIConnectorServer.start(
> >> > RMIConnectorServer.java:427)
> >> >
> >> > ... 4 more
> >> >
> >> > Caused by: java.rmi.ConnectException: Connection refused to host:
> >> > localhost; nested exception is:
> >> >
> >> > java.net.ConnectException: Connection refused
> >> >
> >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> >> > TCPEndpoint.java:619)
> >> >
> >> > at
> >> > sun.rmi.transport.tcp.TCPChannel.createConnection(
> TCPChannel.java:216)
> >> >
> >> > at
> >> > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> >> >
> >> > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
> >> >
> >> > at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
> >> >
> >> > at
> >> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
> >> xt.java:141)
> >> >
> >> > ... 8 more
> >> >
> >> > Caused by: java.net.ConnectException: Connection refused
> >> >
> >> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> >> >
> >> > at
> >> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
> >> etImpl.java:
> >> > 350)
> >> >
> >> > at
> >> > java.net.AbstractPlainSocketImpl.connectToAddress(
> >> > AbstractPlainSocketImpl.java:206)
> >> >
> >> > at
> >> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
> >> Impl.java:188)
> >> >
> >> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >> >
> >> > at java.net.Socket.connect(Socket.java:589)
> >> >
> >> > at java.net.Socket.connect(Socket.java:538)
> >> >
> >> > at java.net.Socket.<init>(Socket.java:434)
> >> >
> >> > at java.net.Socket.<init>(Socket.java:211)
> >> >
> >> > at
> >> > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(
> >> > RMIDirectSocketFactory.java:40)
> >> >
> >> > at
> >> > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(
> >> > RMIMasterSocketFactory.java:148)
> >> >
> >> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> >> > TCPEndpoint.java:613)
> >> >
> >> > ... 13 more
> >> >
> >> > Any idea on what could be the issue?
> >> >
> >> > - Shekar
> >> >
> >>
> >
> >
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
Sorry .. got trigger happy.
Couple of observations:
1. rmi port seem to be random ones Even with task.opts and setting the port
to 1099
2. Do I need to start rmiregistry manually?
3. I dont have any firewall settings.
On Thu, Aug 4, 2016 at 12:14 PM, Shekar Tippur <ct...@gmail.com> wrote:
> Yi
> Thanks for responding.. I have tried diff options for task.opts and
> without it as well.
> I have tried this
>
> task.opts=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099
> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
> -Djava.rmi.server.hostname=localhost
>
>
>
> Here is the property file:
> ##########################################################
>
> # Licensed to the Apache Software Foundation (ASF) under one
>
> # or more contributor license agreements. See the NOTICE file
>
> # distributed with this work for additional information
>
> # regarding copyright ownership. The ASF licenses this file
>
> # to you under the Apache License, Version 2.0 (the
>
> # "License"); you may not use this file except in compliance
>
> # with the License. You may obtain a copy of the License at
>
> #
>
> # http://www.apache.org/licenses/LICENSE-2.0
>
> #
>
> # Unless required by applicable law or agreed to in writing,
>
> # software distributed under the License is distributed on an
>
> # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
>
> # KIND, either express or implied. See the License for the
>
> # specific language governing permissions and limitations
>
> # under the License.
>
>
> job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
>
> job.name=Argos
>
> #job.coordinator.system=kafka
>
> #job.coordinator.replication.factor=2
>
> # YARN
>
> yarn.package.path=http://localhost:8000/argos_aggregate.tar.gz
>
> #task.opts="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7091
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=loc
> alhost"
>
> # Task
>
> task.class=com.project.argos.task.ArgosParserStreamTask
>
> task.inputs=kafka.argos-raw
>
> task.outputs=argos-parser
>
> task.checkpoint.factory=org.apache.samza.checkpoint.kafka.Ka
> fkaCheckpointManagerFactory
>
> task.checkpoint.system=kafka
>
> # Normally, this would be 3, but we have only one broker.
>
> task.checkpoint.replication.factor=2
>
> task.drop.deserialization.errors=true
>
> task.drop.serialization.errors=true
>
> task.ignored.exceptions=org.codehaus.jackson.JsonParseException
>
>
> # Samza10 properties
>
> #systems.kafka.producer.bootstrap.servers=localhost:6667
>
> systems.kafka.producer.bootstrap.servers=host1:6667,host1:6667
>
> job.coordinator.system=kafka
>
>
>
> # Metrics
>
> metrics.reporters=snapshot,jmx
>
> metrics.reporter.snapshot.class=org.apache.samza.metrics.
> reporter.MetricsSnapshotReporterFactory
>
> metrics.reporter.snapshot.stream=kafka.metrics
>
> metrics.reporter.jmx.class=org.apache.samza.metrics.reporter
> .JmxReporterFactory
>
>
> # Serializers
>
> serializers.registry.json.class=org.apache.samza.serializers
> .JsonSerdeFactory
>
> serializers.registry.string.class=org.apache.samza.serialize
> rs.StringSerdeFactory
>
> serializers.registry.metrics.class=org.apache.samza.serializ
> ers.MetricsSnapshotSerdeFactory
>
>
> # Systems
>
> systems.kafka.samza.factory=org.apache.samza.system.kafka.Ka
> fkaSystemFactory
>
> systems.kafka.samza.msg.serde=json
>
>
> systems.kafka.consumer.zookeeper.connect=host1:2181,host2:2181,host3:2181
>
> systems.kafka.consumer.auto.offset.reset=largest
>
> systems.kafka.producer.producer.type=sync
>
> # Normally, we'd set this much higher, but we want things to look snappy
> in the demo.
>
> systems.kafka.producer.batch.num.messages=1
>
> systems.kafka.streams.metrics.samza.msg.serde=metrics
>
> systems.kafka.streams.ArgosRaw.samza.msg.serde=json
>
> systems.kafka.streams.PageViewEvent.samza.msg.serde=json
>
>
>
> # enable wildcard lookup on cache misses, will impact performance
>
> redis.wildcard.lookup=true
>
>
> # Property File
>
> load.property=true
>
>
>
> system.redis=host4
>
> systems.kafka.producer.metadata.broker.list=host1:6667host2:6667
>
> ##########################################################
>
> On Thu, Aug 4, 2016 at 11:19 AM, Yi Pan <ni...@gmail.com> wrote:
>
>> Hi, Shekar,
>>
>> Did you check your firewall configuration? Could you also paste your
>> configuration, especially task.opts?
>>
>> -Yi
>>
>> On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ct...@gmail.com> wrote:
>>
>> > I am trying to submit a Samza job to yarn and I get a error:
>> >
>> > Exception in thread "main" java.io.IOException: Cannot bind to URL
>> > [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailabl
>> eException
>> > [Root exception is java.rmi.ConnectException: Connection refused to
>> host:
>> > localhost; nested exception is:
>> >
>> > java.net.ConnectException: Connection refused]
>> >
>> > at
>> > javax.management.remote.rmi.RMIConnectorServer.newIOException(
>> > RMIConnectorServer.java:827)
>> >
>> > at
>> > javax.management.remote.rmi.RMIConnectorServer.start(
>> > RMIConnectorServer.java:432)
>> >
>> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:89
>> )
>> >
>> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:43
>> )
>> >
>> > at
>> > org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:87)
>> >
>> > at
>> > org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
>> >
>> > Caused by: javax.naming.ServiceUnavailableException [Root exception is
>> > java.rmi.ConnectException: Connection refused to host: localhost; nested
>> > exception is:
>> >
>> > java.net.ConnectException: Connection refused]
>> >
>> > at
>> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
>> xt.java:147)
>> >
>> > at
>> > com.sun.jndi.toolkit.url.GenericURLContext.bind(
>> > GenericURLContext.java:228)
>> >
>> > at javax.naming.InitialContext.bind(InitialContext.java:425)
>> >
>> > at
>> > javax.management.remote.rmi.RMIConnectorServer.bind(
>> > RMIConnectorServer.java:644)
>> >
>> > at
>> > javax.management.remote.rmi.RMIConnectorServer.start(
>> > RMIConnectorServer.java:427)
>> >
>> > ... 4 more
>> >
>> > Caused by: java.rmi.ConnectException: Connection refused to host:
>> > localhost; nested exception is:
>> >
>> > java.net.ConnectException: Connection refused
>> >
>> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
>> > TCPEndpoint.java:619)
>> >
>> > at
>> > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
>> >
>> > at
>> > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
>> >
>> > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
>> >
>> > at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
>> >
>> > at
>> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryConte
>> xt.java:141)
>> >
>> > ... 8 more
>> >
>> > Caused by: java.net.ConnectException: Connection refused
>> >
>> > at java.net.PlainSocketImpl.socketConnect(Native Method)
>> >
>> > at
>> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSock
>> etImpl.java:
>> > 350)
>> >
>> > at
>> > java.net.AbstractPlainSocketImpl.connectToAddress(
>> > AbstractPlainSocketImpl.java:206)
>> >
>> > at
>> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
>> Impl.java:188)
>> >
>> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> >
>> > at java.net.Socket.connect(Socket.java:589)
>> >
>> > at java.net.Socket.connect(Socket.java:538)
>> >
>> > at java.net.Socket.<init>(Socket.java:434)
>> >
>> > at java.net.Socket.<init>(Socket.java:211)
>> >
>> > at
>> > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(
>> > RMIDirectSocketFactory.java:40)
>> >
>> > at
>> > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(
>> > RMIMasterSocketFactory.java:148)
>> >
>> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
>> > TCPEndpoint.java:613)
>> >
>> > ... 13 more
>> >
>> > Any idea on what could be the issue?
>> >
>> > - Shekar
>> >
>>
>
>
Re: Samza yarn job - cannot bind to local host
Posted by Shekar Tippur <ct...@gmail.com>.
Yi
Thanks for responding.. I have tried diff options for task.opts and without
it as well.
I have tried this
task.opts=-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=localhost
Here is the property file:
##########################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
job.factory.class=org.apache.samza.job.yarn.YarnJobFactory
job.name=Argos
#job.coordinator.system=kafka
#job.coordinator.replication.factor=2
# YARN
yarn.package.path=http://localhost:8000/argos_aggregate.tar.gz
#task.opts="-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=7091
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=localhost"
# Task
task.class=com.project.argos.task.ArgosParserStreamTask
task.inputs=kafka.argos-raw
task.outputs=argos-parser
task.checkpoint.factory=org.apache.samza.checkpoint.kafka.
KafkaCheckpointManagerFactory
task.checkpoint.system=kafka
# Normally, this would be 3, but we have only one broker.
task.checkpoint.replication.factor=2
task.drop.deserialization.errors=true
task.drop.serialization.errors=true
task.ignored.exceptions=org.codehaus.jackson.JsonParseException
# Samza10 properties
#systems.kafka.producer.bootstrap.servers=localhost:6667
systems.kafka.producer.bootstrap.servers=host1:6667,host1:6667
job.coordinator.system=kafka
# Metrics
metrics.reporters=snapshot,jmx
metrics.reporter.snapshot.class=org.apache.samza.metrics.reporter.
MetricsSnapshotReporterFactory
metrics.reporter.snapshot.stream=kafka.metrics
metrics.reporter.jmx.class=org.apache.samza.metrics.
reporter.JmxReporterFactory
# Serializers
serializers.registry.json.class=org.apache.samza.
serializers.JsonSerdeFactory
serializers.registry.string.class=org.apache.samza.
serializers.StringSerdeFactory
serializers.registry.metrics.class=org.apache.samza.serializers.
MetricsSnapshotSerdeFactory
# Systems
systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.samza.msg.serde=json
systems.kafka.consumer.zookeeper.connect=host1:2181,host2:2181,host3:2181
systems.kafka.consumer.auto.offset.reset=largest
systems.kafka.producer.producer.type=sync
# Normally, we'd set this much higher, but we want things to look snappy in
the demo.
systems.kafka.producer.batch.num.messages=1
systems.kafka.streams.metrics.samza.msg.serde=metrics
systems.kafka.streams.ArgosRaw.samza.msg.serde=json
systems.kafka.streams.PageViewEvent.samza.msg.serde=json
# enable wildcard lookup on cache misses, will impact performance
redis.wildcard.lookup=true
# Property File
load.property=true
system.redis=host4
systems.kafka.producer.metadata.broker.list=host1:6667host2:6667
##########################################################
On Thu, Aug 4, 2016 at 11:19 AM, Yi Pan <ni...@gmail.com> wrote:
> Hi, Shekar,
>
> Did you check your firewall configuration? Could you also paste your
> configuration, especially task.opts?
>
> -Yi
>
> On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ct...@gmail.com> wrote:
>
> > I am trying to submit a Samza job to yarn and I get a error:
> >
> > Exception in thread "main" java.io.IOException: Cannot bind to URL
> > [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailableException
> > [Root exception is java.rmi.ConnectException: Connection refused to host:
> > localhost; nested exception is:
> >
> > java.net.ConnectException: Connection refused]
> >
> > at
> > javax.management.remote.rmi.RMIConnectorServer.newIOException(
> > RMIConnectorServer.java:827)
> >
> > at
> > javax.management.remote.rmi.RMIConnectorServer.start(
> > RMIConnectorServer.java:432)
> >
> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:89)
> >
> > at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:43)
> >
> > at
> > org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:87)
> >
> > at
> > org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
> >
> > Caused by: javax.naming.ServiceUnavailableException [Root exception is
> > java.rmi.ConnectException: Connection refused to host: localhost; nested
> > exception is:
> >
> > java.net.ConnectException: Connection refused]
> >
> > at
> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:147)
> >
> > at
> > com.sun.jndi.toolkit.url.GenericURLContext.bind(
> > GenericURLContext.java:228)
> >
> > at javax.naming.InitialContext.bind(InitialContext.java:425)
> >
> > at
> > javax.management.remote.rmi.RMIConnectorServer.bind(
> > RMIConnectorServer.java:644)
> >
> > at
> > javax.management.remote.rmi.RMIConnectorServer.start(
> > RMIConnectorServer.java:427)
> >
> > ... 4 more
> >
> > Caused by: java.rmi.ConnectException: Connection refused to host:
> > localhost; nested exception is:
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> > TCPEndpoint.java:619)
> >
> > at
> > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
> >
> > at
> > sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
> >
> > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
> >
> > at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
> >
> > at
> > com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:141)
> >
> > ... 8 more
> >
> > Caused by: java.net.ConnectException: Connection refused
> >
> > at java.net.PlainSocketImpl.socketConnect(Native Method)
> >
> > at
> > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
> > 350)
> >
> > at
> > java.net.AbstractPlainSocketImpl.connectToAddress(
> > AbstractPlainSocketImpl.java:206)
> >
> > at
> > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocket
> Impl.java:188)
> >
> > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >
> > at java.net.Socket.connect(Socket.java:589)
> >
> > at java.net.Socket.connect(Socket.java:538)
> >
> > at java.net.Socket.<init>(Socket.java:434)
> >
> > at java.net.Socket.<init>(Socket.java:211)
> >
> > at
> > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(
> > RMIDirectSocketFactory.java:40)
> >
> > at
> > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(
> > RMIMasterSocketFactory.java:148)
> >
> > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> > TCPEndpoint.java:613)
> >
> > ... 13 more
> >
> > Any idea on what could be the issue?
> >
> > - Shekar
> >
>
Re: Samza yarn job - cannot bind to local host
Posted by Yi Pan <ni...@gmail.com>.
Hi, Shekar,
Did you check your firewall configuration? Could you also paste your
configuration, especially task.opts?
-Yi
On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ct...@gmail.com> wrote:
> I am trying to submit a Samza job to yarn and I get a error:
>
> Exception in thread "main" java.io.IOException: Cannot bind to URL
> [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailableException
> [Root exception is java.rmi.ConnectException: Connection refused to host:
> localhost; nested exception is:
>
> java.net.ConnectException: Connection refused]
>
> at
> javax.management.remote.rmi.RMIConnectorServer.newIOException(
> RMIConnectorServer.java:827)
>
> at
> javax.management.remote.rmi.RMIConnectorServer.start(
> RMIConnectorServer.java:432)
>
> at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:89)
>
> at org.apache.samza.metrics.JmxServer.<init>(JmxServer.scala:43)
>
> at
> org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:87)
>
> at
> org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
>
> Caused by: javax.naming.ServiceUnavailableException [Root exception is
> java.rmi.ConnectException: Connection refused to host: localhost; nested
> exception is:
>
> java.net.ConnectException: Connection refused]
>
> at
> com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:147)
>
> at
> com.sun.jndi.toolkit.url.GenericURLContext.bind(
> GenericURLContext.java:228)
>
> at javax.naming.InitialContext.bind(InitialContext.java:425)
>
> at
> javax.management.remote.rmi.RMIConnectorServer.bind(
> RMIConnectorServer.java:644)
>
> at
> javax.management.remote.rmi.RMIConnectorServer.start(
> RMIConnectorServer.java:427)
>
> ... 4 more
>
> Caused by: java.rmi.ConnectException: Connection refused to host:
> localhost; nested exception is:
>
> java.net.ConnectException: Connection refused
>
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> TCPEndpoint.java:619)
>
> at
> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
>
> at
> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
>
> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342)
>
> at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
>
> at
> com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:141)
>
> ... 8 more
>
> Caused by: java.net.ConnectException: Connection refused
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
> 350)
>
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
>
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at java.net.Socket.connect(Socket.java:538)
>
> at java.net.Socket.<init>(Socket.java:434)
>
> at java.net.Socket.<init>(Socket.java:211)
>
> at
> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(
> RMIDirectSocketFactory.java:40)
>
> at
> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(
> RMIMasterSocketFactory.java:148)
>
> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(
> TCPEndpoint.java:613)
>
> ... 13 more
>
> Any idea on what could be the issue?
>
> - Shekar
>