You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Kwan Chong Tan <kc...@gmail.com> on 2016/07/27 06:48:39 UTC
Storm Topology Hostname Issues
Hi all,
I have Metron deployed on a single node on AWS. Recently I tried updating
the hostname to use AWS private DNS instead of the public DNS (which
changes with each reboot). I think I have got most of the services working
after updating the hostname but I have still have some issues with Storm.
A sample of the storm worker logs are copied below. In particular, the
o.a.h.i.Client seems to be still referring to the old public EC2 domain
name but I have been unable to figure out where that config is specified.
Could someone assist in pointing me to where that particular variable is
stored?
Thanks,
KC
2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] Deleted
partition managers: []
2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] New partition
managers: []
2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] Finished
refreshing
2016-07-27 06:41:22.253 b.s.m.n.Server [INFO] Getting metrics for server on
port 6704
2016-07-27 06:41:24.037 o.a.h.i.Client [INFO] Retrying connect to server:
ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
Already tried 32 time(s); maxRetries=45
2016-07-27 06:41:44.058 o.a.h.i.Client [INFO] Retrying connect to server:
ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
Already tried 33 time(s); maxRetries=45
2016-07-27 06:42:04.078 o.a.h.i.Client [INFO] Retrying connect to server:
ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
Already tried 34 time(s); maxRetries=45
2016-07-27 06:42:21.626 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
partition manager connections
2016-07-27 06:42:21.627 s.k.DynamicBrokersReader [INFO] Read partition info
from zookeeper:
GlobalPartitionInformation{partitionMap={0=ip-10-0-0-21.us-west-2.compute.internal:6667}}
2016-07-27 06:42:21.627 s.k.KafkaUtils [INFO] Task [1/1] assigned
[Partition{host=ip-10-0-0-21.us-west-2.compute.internal:6667, partition=0}]
2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] Deleted
partition managers: []
2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] New partition
managers: []
2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] Finished
refreshing
2016-07-27 06:42:22.254 b.s.m.n.Server [INFO] Getting metrics for server on
port 6704
2016-07-27 06:42:24.104 o.a.h.i.Client [INFO] Retrying connect to server:
ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
Already tried 35 time(s); maxRetries=45
2016-07-27 06:42:44.121 o.a.h.i.Client [INFO] Retrying connect to server:
ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
Already tried 36 time(s); maxRetries=45
2016-07-27 06:43:04.139 o.a.h.i.Client [INFO] Retrying connect to server:
ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
Already tried 37 time(s); maxRetries=45
2016-07-27 06:43:21.629 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
partition manager connections
2016-07-27 06:43:21.630 s.k.DynamicBrokersReader [INFO] Read partition info
from zookeeper:
GlobalPartitionInformation{partitionMap={0=ip-10-0-0-21.us-west-2.compute.internal:6667}}
2016-07-27 06:43:21.631 s.k.KafkaUtils [INFO] Task [1/1] assigned
[Partition{host=ip-10-0-0-21.us-west-2.compute.internal:6667, partition=0}]
Re: Storm Topology Hostname Issues
Posted by Kwan Chong Tan <kc...@gmail.com>.
Ok I realized the *-site.xml files were inserted directly into the topology
jars themselves so that's where the problem probably lies....
On Thu, Jul 28, 2016 at 4:40 PM, Kwan Chong Tan <kc...@gmail.com> wrote:
> Hi all,
>
> I have narrowed this down to the GrokParser trying to retrieve the Grok
> Pattern from HDFS but for some reason it's still referencing the invalid
> ec2-54-213-184 hostname. Does anyone know where Configuration is getting
> the Hadoop FS url from? All the core-default.xml and core-site.xml files
> are referring to the correct internal hostname so I'm confused why the
> GrokParser is still referring to the invalid hostname. Any assistance would
> be much appreciated. Thanks!
>
> Regards,
> KC
>
> public InputStream openInputStream(String streamName) throws IOException {
> FileSystem fs = FileSystem.get(new Configuration());
> Path path = new Path(streamName);
> if(fs.exists(path)) {
> return fs.open(path);
> } else {
> return getClass().getResourceAsStream(streamName);
> }
> }
>
>
> java.lang.RuntimeException: Grok parser Error: Call From
> ip-10-0-0-21.us-west-2.compute.internal/10.0.0.21 to
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com:8020 failed on socket
> timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000
> millis timeout while waiting for channel to be ready for connect. ch :
> java.nio.channels.SocketChannel[connection-pending
> remote=ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020];
> For more details see: http://wiki.apache.org/hadoop/SocketTimeout at
> org.apache.metron.parsers.GrokParser.init(GrokParser.java:127) at
> org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:90) at
> backtype.storm.daemon.executor$fn__7143$fn__7156.invoke(executor.clj:751)
> at backtype.storm.util$async_loop$fn__1985.invoke(util.clj:477) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.net.ConnectTimeoutException: Call From
> ip-10-0-0-21.us-west-2.compute.internal/10.0.0.21 to
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com:8020 failed on socket
> <http://ip-10-0-0-21.us-west-2.compute.internal:8000/log?file=squid-11-1469690590-worker-6704.log>
>
> On Wed, Jul 27, 2016 at 2:48 PM, Kwan Chong Tan <kc...@gmail.com> wrote:
>
>> Hi all,
>>
>> I have Metron deployed on a single node on AWS. Recently I tried updating
>> the hostname to use AWS private DNS instead of the public DNS (which
>> changes with each reboot). I think I have got most of the services working
>> after updating the hostname but I have still have some issues with Storm.
>>
>> A sample of the storm worker logs are copied below. In particular, the
>> o.a.h.i.Client seems to be still referring to the old public EC2 domain
>> name but I have been unable to figure out where that config is specified.
>> Could someone assist in pointing me to where that particular variable is
>> stored?
>>
>> Thanks,
>> KC
>>
>> 2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] Deleted
>> partition managers: []
>> 2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] New partition
>> managers: []
>> 2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] Finished
>> refreshing
>> 2016-07-27 06:41:22.253 b.s.m.n.Server [INFO] Getting metrics for server
>> on port 6704
>> 2016-07-27 06:41:24.037 o.a.h.i.Client [INFO] Retrying connect to server:
>> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
>> Already tried 32 time(s); maxRetries=45
>> 2016-07-27 06:41:44.058 o.a.h.i.Client [INFO] Retrying connect to server:
>> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
>> Already tried 33 time(s); maxRetries=45
>> 2016-07-27 06:42:04.078 o.a.h.i.Client [INFO] Retrying connect to server:
>> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
>> Already tried 34 time(s); maxRetries=45
>> 2016-07-27 06:42:21.626 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
>> partition manager connections
>> 2016-07-27 06:42:21.627 s.k.DynamicBrokersReader [INFO] Read partition
>> info from zookeeper:
>> GlobalPartitionInformation{partitionMap={0=ip-10-0-0-21.us-west-2.compute.internal:6667}}
>> 2016-07-27 06:42:21.627 s.k.KafkaUtils [INFO] Task [1/1] assigned
>> [Partition{host=ip-10-0-0-21.us-west-2.compute.internal:6667, partition=0}]
>> 2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] Deleted
>> partition managers: []
>> 2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] New partition
>> managers: []
>> 2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] Finished
>> refreshing
>> 2016-07-27 06:42:22.254 b.s.m.n.Server [INFO] Getting metrics for server
>> on port 6704
>> 2016-07-27 06:42:24.104 o.a.h.i.Client [INFO] Retrying connect to server:
>> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
>> Already tried 35 time(s); maxRetries=45
>> 2016-07-27 06:42:44.121 o.a.h.i.Client [INFO] Retrying connect to server:
>> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
>> Already tried 36 time(s); maxRetries=45
>> 2016-07-27 06:43:04.139 o.a.h.i.Client [INFO] Retrying connect to server:
>> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
>> Already tried 37 time(s); maxRetries=45
>> 2016-07-27 06:43:21.629 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
>> partition manager connections
>> 2016-07-27 06:43:21.630 s.k.DynamicBrokersReader [INFO] Read partition
>> info from zookeeper:
>> GlobalPartitionInformation{partitionMap={0=ip-10-0-0-21.us-west-2.compute.internal:6667}}
>> 2016-07-27 06:43:21.631 s.k.KafkaUtils [INFO] Task [1/1] assigned
>> [Partition{host=ip-10-0-0-21.us-west-2.compute.internal:6667, partition=0}]
>>
>>
>
Re: Storm Topology Hostname Issues
Posted by Kwan Chong Tan <kc...@gmail.com>.
Hi all,
I have narrowed this down to the GrokParser trying to retrieve the Grok
Pattern from HDFS but for some reason it's still referencing the invalid
ec2-54-213-184 hostname. Does anyone know where Configuration is getting
the Hadoop FS url from? All the core-default.xml and core-site.xml files
are referring to the correct internal hostname so I'm confused why the
GrokParser is still referring to the invalid hostname. Any assistance would
be much appreciated. Thanks!
Regards,
KC
public InputStream openInputStream(String streamName) throws IOException {
FileSystem fs = FileSystem.get(new Configuration());
Path path = new Path(streamName);
if(fs.exists(path)) {
return fs.open(path);
} else {
return getClass().getResourceAsStream(streamName);
}
}
java.lang.RuntimeException: Grok parser Error: Call From
ip-10-0-0-21.us-west-2.compute.internal/10.0.0.21 to
ec2-54-213-184-142.us-west-2.compute.amazonaws.com:8020 failed on socket
timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000
millis timeout while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending
remote=ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020];
For more details see: http://wiki.apache.org/hadoop/SocketTimeout at
org.apache.metron.parsers.GrokParser.init(GrokParser.java:127) at
org.apache.metron.parsers.bolt.ParserBolt.prepare(ParserBolt.java:90) at
backtype.storm.daemon.executor$fn__7143$fn__7156.invoke(executor.clj:751)
at backtype.storm.util$async_loop$fn__1985.invoke(util.clj:477) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.net.ConnectTimeoutException: Call From
ip-10-0-0-21.us-west-2.compute.internal/10.0.0.21 to
ec2-54-213-184-142.us-west-2.compute.amazonaws.com:8020 failed on socket
<http://ip-10-0-0-21.us-west-2.compute.internal:8000/log?file=squid-11-1469690590-worker-6704.log>
On Wed, Jul 27, 2016 at 2:48 PM, Kwan Chong Tan <kc...@gmail.com> wrote:
> Hi all,
>
> I have Metron deployed on a single node on AWS. Recently I tried updating
> the hostname to use AWS private DNS instead of the public DNS (which
> changes with each reboot). I think I have got most of the services working
> after updating the hostname but I have still have some issues with Storm.
>
> A sample of the storm worker logs are copied below. In particular, the
> o.a.h.i.Client seems to be still referring to the old public EC2 domain
> name but I have been unable to figure out where that config is specified.
> Could someone assist in pointing me to where that particular variable is
> stored?
>
> Thanks,
> KC
>
> 2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] Deleted
> partition managers: []
> 2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] New partition
> managers: []
> 2016-07-27 06:41:21.625 s.k.ZkCoordinator [INFO] Task [1/1] Finished
> refreshing
> 2016-07-27 06:41:22.253 b.s.m.n.Server [INFO] Getting metrics for server
> on port 6704
> 2016-07-27 06:41:24.037 o.a.h.i.Client [INFO] Retrying connect to server:
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
> Already tried 32 time(s); maxRetries=45
> 2016-07-27 06:41:44.058 o.a.h.i.Client [INFO] Retrying connect to server:
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
> Already tried 33 time(s); maxRetries=45
> 2016-07-27 06:42:04.078 o.a.h.i.Client [INFO] Retrying connect to server:
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
> Already tried 34 time(s); maxRetries=45
> 2016-07-27 06:42:21.626 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
> partition manager connections
> 2016-07-27 06:42:21.627 s.k.DynamicBrokersReader [INFO] Read partition
> info from zookeeper:
> GlobalPartitionInformation{partitionMap={0=ip-10-0-0-21.us-west-2.compute.internal:6667}}
> 2016-07-27 06:42:21.627 s.k.KafkaUtils [INFO] Task [1/1] assigned
> [Partition{host=ip-10-0-0-21.us-west-2.compute.internal:6667, partition=0}]
> 2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] Deleted
> partition managers: []
> 2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] New partition
> managers: []
> 2016-07-27 06:42:21.628 s.k.ZkCoordinator [INFO] Task [1/1] Finished
> refreshing
> 2016-07-27 06:42:22.254 b.s.m.n.Server [INFO] Getting metrics for server
> on port 6704
> 2016-07-27 06:42:24.104 o.a.h.i.Client [INFO] Retrying connect to server:
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
> Already tried 35 time(s); maxRetries=45
> 2016-07-27 06:42:44.121 o.a.h.i.Client [INFO] Retrying connect to server:
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
> Already tried 36 time(s); maxRetries=45
> 2016-07-27 06:43:04.139 o.a.h.i.Client [INFO] Retrying connect to server:
> ec2-54-213-184-142.us-west-2.compute.amazonaws.com/54.213.184.142:8020.
> Already tried 37 time(s); maxRetries=45
> 2016-07-27 06:43:21.629 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
> partition manager connections
> 2016-07-27 06:43:21.630 s.k.DynamicBrokersReader [INFO] Read partition
> info from zookeeper:
> GlobalPartitionInformation{partitionMap={0=ip-10-0-0-21.us-west-2.compute.internal:6667}}
> 2016-07-27 06:43:21.631 s.k.KafkaUtils [INFO] Task [1/1] assigned
> [Partition{host=ip-10-0-0-21.us-west-2.compute.internal:6667, partition=0}]
>
>