You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by jova <jo...@gmail.com> on 2014/10/22 05:21:02 UTC
Storm Cluster on Linux and Wndows
Has anyone successfully run a storm cluster with Linux and Windows boxes?
I am attempting this, by running nimbus on a linux box and have a
supervisor on a windows box.
Everything seems fine until I submit a topology jar and the supervisor node
attempts to download the topology jar.
Here is the error I get
2014-10-21 23:09:40 b.s.d.supervisor [INFO] Downloading code for storm id
JTopology-1-1413947365 from
/home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
org.apache.thrift7.transport.TTransportException: null
My storm.yaml file on my windows box (supervisor node) stores the storm
data in a different directory.
Anyone with any ideas what can be causing this?
--
Jovaughn Lockridge
{developer : Github <https://github.com/settings/profile> }
{professional : LinkedIn <https://www.linkedin.com/profile/view?id=20227395>
}
{social : twitter <https://twitter.com/JovaCoder> }
Re: Storm Cluster on Linux and Wndows
Posted by Jovaughn Lockridge <jo...@gmail.com>.
I ran my same storm.yaml configuration (minus the minor changes to the
directory structure) and it works. I believe this a bug if the cluster
was intended to include windows and linux nodes. Where can I submit a JIRA
for this?
On Thu, Oct 23, 2014 at 1:00 AM, jova <jo...@gmail.com> wrote:
> The actual exception would help
>
> 2014-10-23 00:52:00 o.a.t.s.TNonblockingServer [ERROR] Unexpected
> exception while invoking!
> java.io.FileNotFoundException:
> /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology_Prod_Cluster-2-1414039471\stormjar.jar
> (No such file or direc
> tory)
> at java.io.FileInputStream.open(Native Method) ~[na:1.8.0_05]
>
>
>
> On Thu, Oct 23, 2014 at 12:58 AM, jova <jo...@gmail.com> wrote:
>
>> So I checked my nimbus.log and I see this exeception
>>
>>
>> * /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology_Prod_Cluster-2-1414039471\stormjar.jar*
>>
>> Notice the last slash is a back slash. I assume this is caused by
>> Windows in some way. Does anyone have any suggestions on how to fix?
>>
>> On Wed, Oct 22, 2014 at 8:27 AM, jova <jo...@gmail.com> wrote:
>>
>>> I have been using the default values for those fields.
>>>
>>> I suspect that the "storm.local.dir" must be the same on the nimbus and
>>> supervisor nodes.
>>>
>>> I can take a look at the code or do an empirical analysis later today to
>>> verify.
>>>
>>> On Oct 22, 2014 12:08 AM, "Harsha" <st...@harsha.io> wrote:
>>> >
>>> > from the logs you posted it seems to be that the supervisor running on
>>> the windows machine is not able to connect to your nimbus. Make sure you
>>> configured the storm.yaml with right nimbus.host and nimbus.thrift.port
>>> which should be whatever you used on the linux box and also zookeeper host.
>>> Incase if you are trying out storm you can run supervisor on the same host
>>> without the need for supervisor running on a different os and host.
>>> > -Harsha
>>> >
>>> >
>>> > On Tue, Oct 21, 2014, at 08:49 PM, jova wrote:
>>> >>
>>> >> Yes I can ping my linux box from windows. I can also bring up the
>>> storm ui from my windows box
>>> >>
>>> >> I am running 2 different OS because I have windows 8 on one machine
>>> and it is not easy to configure a dual boot (linux/windows) anymore (from
>>> what I read). On my linux box I already configured everything (kafka,
>>> zookeeper, and storm) in Linux I did not have the energy to reconfigure
>>> under windows.
>>> >>
>>> >> On Tue, Oct 21, 2014 at 11:40 PM, Harsha <st...@harsha.io> wrote:
>>> >>>
>>> >>>
>>> >>>
>>> >>> Are you able to ping your linux host from windows. any specific
>>> reason to run a distributed cluster on different OS.
>>> >>> Thanks,
>>> >>> Harsha
>>> >>>
>>> >>> On Tue, Oct 21, 2014, at 08:21 PM, jova wrote:
>>> >>>>
>>> >>>> Has anyone successfully run a storm cluster with Linux and Windows
>>> boxes?
>>> >>>> I am attempting this, by running nimbus on a linux box and have a
>>> supervisor on a windows box.
>>> >>>> Everything seems fine until I submit a topology jar and the
>>> supervisor node attempts to download the topology jar.
>>> >>>>
>>> >>>> Here is the error I get
>>> >>>>
>>> >>>> 2014-10-21 23:09:40 b.s.d.supervisor [INFO] Downloading code for
>>> storm id JTopology-1-1413947365 from
>>> /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
>>> >>>> 2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
>>> >>>> org.apache.thrift7.transport.TTransportException: null
>>> >>>>
>>> >>>> My storm.yaml file on my windows box (supervisor node) stores the
>>> storm data in a different directory.
>>> >>>>
>>> >>>> Anyone with any ideas what can be causing this?
>>> >>>>
>>> >>>> --
>>> >>>> Jovaughn Lockridge
>>> >>>>
>>> >>>> {developer : Github }
>>> >>>> {professional : LinkedIn }
>>> >>>> {social : twitter }
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Jovaughn Lockridge
>>> >>
>>> >> {developer : Github }
>>> >> {professional : LinkedIn }
>>> >> {social : twitter }
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Jovaughn Lockridge
>>
>> {developer : Github <https://github.com/settings/profile> }
>> {professional : LinkedIn
>> <https://www.linkedin.com/profile/view?id=20227395> }
>> {social : twitter <https://twitter.com/JovaCoder> }
>>
>
>
>
> --
> Jovaughn Lockridge
>
> {developer : Github <https://github.com/settings/profile> }
> {professional : LinkedIn
> <https://www.linkedin.com/profile/view?id=20227395> }
> {social : twitter <https://twitter.com/JovaCoder> }
>
--
Jovaughn Lockridge
{developer : Github <https://github.com/settings/profile> }
{professional : LinkedIn <https://www.linkedin.com/profile/view?id=20227395>
}
{social : twitter <https://twitter.com/JovaCoder> }
Re: Storm Cluster on Linux and Wndows
Posted by jova <jo...@gmail.com>.
The actual exception would help
2014-10-23 00:52:00 o.a.t.s.TNonblockingServer [ERROR] Unexpected exception
while invoking!
java.io.FileNotFoundException:
/home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology_Prod_Cluster-2-1414039471\stormjar.jar
(No such file or direc
tory)
at java.io.FileInputStream.open(Native Method) ~[na:1.8.0_05]
On Thu, Oct 23, 2014 at 12:58 AM, jova <jo...@gmail.com> wrote:
> So I checked my nimbus.log and I see this exeception
>
>
> * /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology_Prod_Cluster-2-1414039471\stormjar.jar*
>
> Notice the last slash is a back slash. I assume this is caused by Windows
> in some way. Does anyone have any suggestions on how to fix?
>
> On Wed, Oct 22, 2014 at 8:27 AM, jova <jo...@gmail.com> wrote:
>
>> I have been using the default values for those fields.
>>
>> I suspect that the "storm.local.dir" must be the same on the nimbus and
>> supervisor nodes.
>>
>> I can take a look at the code or do an empirical analysis later today to
>> verify.
>>
>> On Oct 22, 2014 12:08 AM, "Harsha" <st...@harsha.io> wrote:
>> >
>> > from the logs you posted it seems to be that the supervisor running on
>> the windows machine is not able to connect to your nimbus. Make sure you
>> configured the storm.yaml with right nimbus.host and nimbus.thrift.port
>> which should be whatever you used on the linux box and also zookeeper host.
>> Incase if you are trying out storm you can run supervisor on the same host
>> without the need for supervisor running on a different os and host.
>> > -Harsha
>> >
>> >
>> > On Tue, Oct 21, 2014, at 08:49 PM, jova wrote:
>> >>
>> >> Yes I can ping my linux box from windows. I can also bring up the
>> storm ui from my windows box
>> >>
>> >> I am running 2 different OS because I have windows 8 on one machine
>> and it is not easy to configure a dual boot (linux/windows) anymore (from
>> what I read). On my linux box I already configured everything (kafka,
>> zookeeper, and storm) in Linux I did not have the energy to reconfigure
>> under windows.
>> >>
>> >> On Tue, Oct 21, 2014 at 11:40 PM, Harsha <st...@harsha.io> wrote:
>> >>>
>> >>>
>> >>>
>> >>> Are you able to ping your linux host from windows. any specific
>> reason to run a distributed cluster on different OS.
>> >>> Thanks,
>> >>> Harsha
>> >>>
>> >>> On Tue, Oct 21, 2014, at 08:21 PM, jova wrote:
>> >>>>
>> >>>> Has anyone successfully run a storm cluster with Linux and Windows
>> boxes?
>> >>>> I am attempting this, by running nimbus on a linux box and have a
>> supervisor on a windows box.
>> >>>> Everything seems fine until I submit a topology jar and the
>> supervisor node attempts to download the topology jar.
>> >>>>
>> >>>> Here is the error I get
>> >>>>
>> >>>> 2014-10-21 23:09:40 b.s.d.supervisor [INFO] Downloading code for
>> storm id JTopology-1-1413947365 from
>> /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
>> >>>> 2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
>> >>>> org.apache.thrift7.transport.TTransportException: null
>> >>>>
>> >>>> My storm.yaml file on my windows box (supervisor node) stores the
>> storm data in a different directory.
>> >>>>
>> >>>> Anyone with any ideas what can be causing this?
>> >>>>
>> >>>> --
>> >>>> Jovaughn Lockridge
>> >>>>
>> >>>> {developer : Github }
>> >>>> {professional : LinkedIn }
>> >>>> {social : twitter }
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Jovaughn Lockridge
>> >>
>> >> {developer : Github }
>> >> {professional : LinkedIn }
>> >> {social : twitter }
>> >
>> >
>>
>
>
>
> --
> Jovaughn Lockridge
>
> {developer : Github <https://github.com/settings/profile> }
> {professional : LinkedIn
> <https://www.linkedin.com/profile/view?id=20227395> }
> {social : twitter <https://twitter.com/JovaCoder> }
>
--
Jovaughn Lockridge
{developer : Github <https://github.com/settings/profile> }
{professional : LinkedIn <https://www.linkedin.com/profile/view?id=20227395>
}
{social : twitter <https://twitter.com/JovaCoder> }
Re: Storm Cluster on Linux and Wndows
Posted by jova <jo...@gmail.com>.
So I checked my nimbus.log and I see this exeception
* /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology_Prod_Cluster-2-1414039471\stormjar.jar*
Notice the last slash is a back slash. I assume this is caused by Windows
in some way. Does anyone have any suggestions on how to fix?
On Wed, Oct 22, 2014 at 8:27 AM, jova <jo...@gmail.com> wrote:
> I have been using the default values for those fields.
>
> I suspect that the "storm.local.dir" must be the same on the nimbus and
> supervisor nodes.
>
> I can take a look at the code or do an empirical analysis later today to
> verify.
>
> On Oct 22, 2014 12:08 AM, "Harsha" <st...@harsha.io> wrote:
> >
> > from the logs you posted it seems to be that the supervisor running on
> the windows machine is not able to connect to your nimbus. Make sure you
> configured the storm.yaml with right nimbus.host and nimbus.thrift.port
> which should be whatever you used on the linux box and also zookeeper host.
> Incase if you are trying out storm you can run supervisor on the same host
> without the need for supervisor running on a different os and host.
> > -Harsha
> >
> >
> > On Tue, Oct 21, 2014, at 08:49 PM, jova wrote:
> >>
> >> Yes I can ping my linux box from windows. I can also bring up the
> storm ui from my windows box
> >>
> >> I am running 2 different OS because I have windows 8 on one machine and
> it is not easy to configure a dual boot (linux/windows) anymore (from what
> I read). On my linux box I already configured everything (kafka,
> zookeeper, and storm) in Linux I did not have the energy to reconfigure
> under windows.
> >>
> >> On Tue, Oct 21, 2014 at 11:40 PM, Harsha <st...@harsha.io> wrote:
> >>>
> >>>
> >>>
> >>> Are you able to ping your linux host from windows. any specific reason
> to run a distributed cluster on different OS.
> >>> Thanks,
> >>> Harsha
> >>>
> >>> On Tue, Oct 21, 2014, at 08:21 PM, jova wrote:
> >>>>
> >>>> Has anyone successfully run a storm cluster with Linux and Windows
> boxes?
> >>>> I am attempting this, by running nimbus on a linux box and have a
> supervisor on a windows box.
> >>>> Everything seems fine until I submit a topology jar and the
> supervisor node attempts to download the topology jar.
> >>>>
> >>>> Here is the error I get
> >>>>
> >>>> 2014-10-21 23:09:40 b.s.d.supervisor [INFO] Downloading code for
> storm id JTopology-1-1413947365 from
> /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
> >>>> 2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
> >>>> org.apache.thrift7.transport.TTransportException: null
> >>>>
> >>>> My storm.yaml file on my windows box (supervisor node) stores the
> storm data in a different directory.
> >>>>
> >>>> Anyone with any ideas what can be causing this?
> >>>>
> >>>> --
> >>>> Jovaughn Lockridge
> >>>>
> >>>> {developer : Github }
> >>>> {professional : LinkedIn }
> >>>> {social : twitter }
> >>>
> >>>
> >>
> >>
> >>
> >>
> >> --
> >> Jovaughn Lockridge
> >>
> >> {developer : Github }
> >> {professional : LinkedIn }
> >> {social : twitter }
> >
> >
>
--
Jovaughn Lockridge
{developer : Github <https://github.com/settings/profile> }
{professional : LinkedIn <https://www.linkedin.com/profile/view?id=20227395>
}
{social : twitter <https://twitter.com/JovaCoder> }
Re: Storm Cluster on Linux and Wndows
Posted by jova <jo...@gmail.com>.
I have been using the default values for those fields.
I suspect that the "storm.local.dir" must be the same on the nimbus and
supervisor nodes.
I can take a look at the code or do an empirical analysis later today to
verify.
On Oct 22, 2014 12:08 AM, "Harsha" <st...@harsha.io> wrote:
>
> from the logs you posted it seems to be that the supervisor running on
the windows machine is not able to connect to your nimbus. Make sure you
configured the storm.yaml with right nimbus.host and nimbus.thrift.port
which should be whatever you used on the linux box and also zookeeper host.
Incase if you are trying out storm you can run supervisor on the same host
without the need for supervisor running on a different os and host.
> -Harsha
>
>
> On Tue, Oct 21, 2014, at 08:49 PM, jova wrote:
>>
>> Yes I can ping my linux box from windows. I can also bring up the storm
ui from my windows box
>>
>> I am running 2 different OS because I have windows 8 on one machine and
it is not easy to configure a dual boot (linux/windows) anymore (from what
I read). On my linux box I already configured everything (kafka,
zookeeper, and storm) in Linux I did not have the energy to reconfigure
under windows.
>>
>> On Tue, Oct 21, 2014 at 11:40 PM, Harsha <st...@harsha.io> wrote:
>>>
>>>
>>>
>>> Are you able to ping your linux host from windows. any specific reason
to run a distributed cluster on different OS.
>>> Thanks,
>>> Harsha
>>>
>>> On Tue, Oct 21, 2014, at 08:21 PM, jova wrote:
>>>>
>>>> Has anyone successfully run a storm cluster with Linux and Windows
boxes?
>>>> I am attempting this, by running nimbus on a linux box and have a
supervisor on a windows box.
>>>> Everything seems fine until I submit a topology jar and the supervisor
node attempts to download the topology jar.
>>>>
>>>> Here is the error I get
>>>>
>>>> 2014-10-21 23:09:40 b.s.d.supervisor [INFO] Downloading code for storm
id JTopology-1-1413947365 from
/home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
>>>> 2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
>>>> org.apache.thrift7.transport.TTransportException: null
>>>>
>>>> My storm.yaml file on my windows box (supervisor node) stores the
storm data in a different directory.
>>>>
>>>> Anyone with any ideas what can be causing this?
>>>>
>>>> --
>>>> Jovaughn Lockridge
>>>>
>>>> {developer : Github }
>>>> {professional : LinkedIn }
>>>> {social : twitter }
>>>
>>>
>>
>>
>>
>>
>> --
>> Jovaughn Lockridge
>>
>> {developer : Github }
>> {professional : LinkedIn }
>> {social : twitter }
>
>
Re: Storm Cluster on Linux and Wndows
Posted by Harsha <st...@harsha.io>.
from the logs you posted it seems to be that the supervisor running on
the windows machine is not able to connect to your nimbus. Make sure you
configured the storm.yaml with right nimbus.host and nimbus.thrift.port
which should be whatever you used on the linux box and also zookeeper
host. Incase if you are trying out storm you can run supervisor on the
same host without the need for supervisor running on a different os and
host. -Harsha
On Tue, Oct 21, 2014, at 08:49 PM, jova wrote:
> Yes I can ping my linux box from windows. I can also bring up the
> storm ui from my windows box
>
> I am running 2 different OS because I have windows 8 on one machine
> and it is not easy to configure a dual boot (linux/windows) anymore
> (from what I read). On my linux box I already configured everything
> (kafka, zookeeper, and storm) in Linux I did not have the energy to
> reconfigure under windows.
>
> On Tue, Oct 21, 2014 at 11:40 PM, Harsha <st...@harsha.io> wrote:
>> __
>>
>> Are you able to ping your linux host from windows. any specific
>> reason to run a distributed cluster on different OS. Thanks, Harsha
>>
>> On Tue, Oct 21, 2014, at 08:21 PM, jova wrote:
>>> Has anyone successfully run a storm cluster with Linux and Windows
>>> boxes? I am attempting this, by running nimbus on a linux box and
>>> have a supervisor on a windows box. Everything seems fine until I
>>> submit a topology jar and the supervisor node attempts to download
>>> the topology jar.
>>>
>>> Here is the error I get
>>>
>>> 2014-10-21 23:09:40 b.s.d.supervisor [INFO] Downloading code for
>>> storm id JTopology-1-1413947365 from
>>> /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
>>> 2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
>>> org.apache.thrift7.transport.TTransportException: null
>>>
>>> My storm.yaml file on my windows box (supervisor node) stores the
>>> storm data in a different directory.
>>>
>>> Anyone with any ideas what can be causing this?
>>>
>>> --
>>> Jovaughn Lockridge
>>>
>>> {developer : Github[1] } {professional : LinkedIn[2] } {social :
>>> twitter[3] }
>>
>
>
>
> --
> Jovaughn Lockridge
>
> {developer : Github[4] } {professional : LinkedIn[5] } {social :
> twitter[6] }
Links:
1. https://github.com/settings/profile
2. https://www.linkedin.com/profile/view?id=20227395
3. https://twitter.com/JovaCoder
4. https://github.com/settings/profile
5. https://www.linkedin.com/profile/view?id=20227395
6. https://twitter.com/JovaCoder
Re: Storm Cluster on Linux and Wndows
Posted by jova <jo...@gmail.com>.
Yes I can ping my linux box from windows. I can also bring up the storm ui
from my windows box
I am running 2 different OS because I have windows 8 on one machine and it
is not easy to configure a dual boot (linux/windows) anymore (from what I
read). On my linux box I already configured everything (kafka, zookeeper,
and storm) in Linux I did not have the energy to reconfigure under windows.
On Tue, Oct 21, 2014 at 11:40 PM, Harsha <st...@harsha.io> wrote:
>
> Are you able to ping your linux host from windows. any specific reason to
> run a distributed cluster on different OS.
> Thanks,
> Harsha
>
> On Tue, Oct 21, 2014, at 08:21 PM, jova wrote:
>
> Has anyone successfully run a storm cluster with Linux and Windows boxes?
> I am attempting this, by running nimbus on a linux box and have a
> supervisor on a windows box.
> Everything seems fine until I submit a topology jar and the supervisor
> node attempts to download the topology jar.
>
> Here is the error I get
>
> 2014-10-21 23:09:40 b.s.d.supervisor [INFO] Downloading code for storm id
> JTopology-1-1413947365 from
> /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
> 2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
> org.apache.thrift7.transport.TTransportException: null
>
> My storm.yaml file on my windows box (supervisor node) stores the storm
> data in a different directory.
>
> Anyone with any ideas what can be causing this?
>
> --
> Jovaughn Lockridge
>
> {developer : Github <https://github.com/settings/profile> }
> {professional : LinkedIn
> <https://www.linkedin.com/profile/view?id=20227395> }
> {social : twitter <https://twitter.com/JovaCoder> }
>
>
>
--
Jovaughn Lockridge
{developer : Github <https://github.com/settings/profile> }
{professional : LinkedIn <https://www.linkedin.com/profile/view?id=20227395>
}
{social : twitter <https://twitter.com/JovaCoder> }
Re: Storm Cluster on Linux and Wndows
Posted by Harsha <st...@harsha.io>.
� Are you able to ping your linux host from windows. any specific reason
to run a distributed cluster on different OS. Thanks, Harsha � On Tue,
Oct 21, 2014, at 08:21 PM, jova wrote:
> Has anyone successfully run a storm cluster with Linux and Windows
> boxes? I am attempting this, by running nimbus on a linux box and
> have a supervisor on a windows box. � Everything seems fine until I
> submit a topology jar and the supervisor node attempts to download
> the topology jar.� � Here is the error I get� � 2014-10-21 23:09:40
> b.s.d.supervisor [INFO] Downloading code for storm id
> JTopology-1-1413947365 from
> /home/jova/Apps/apache-storm-0.9.2-incubating/data/nimbus/stormdist/JTopology-1-1413947365
> 2014-10-21 23:09:40 b.s.event [ERROR] Error when processing event
> org.apache.thrift7.transport.TTransportException: null � My
> storm.yaml file on my windows box (supervisor node) stores the storm
> data in a different directory. � � Anyone with any ideas what can be
> causing this? �
> --
> Jovaughn Lockridge � {developer : Github[1]�} {professional
> :�LinkedIn[2]�} {social :�twitter[3]�}
�
Links:
1. https://github.com/settings/profile
2. https://www.linkedin.com/profile/view?id=20227395
3. https://twitter.com/JovaCoder