You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by S G <sg...@gmail.com> on 2016/09/12 19:24:26 UTC
Flux exception while upgrading to Storm 1.0.2
Hi,
We are upgrading to Storm 1.0.2 and have done all the package name changes.
grep -ri backtype in our code repo returns nothing.
Yet on deploying the jar to Storm, following error is seen:
storm/storm_installed/apache-storm-1.0.2/bin/storm \
jar \
$jarFile \
org.apache.storm.flux.Flux \
$yamlFileTarget \
--remote
+- Apache Storm -+
+- data FLow User eXperience -+
Version: 1.0.2
Parsing file: topology-config.yaml
333 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
335 [main] INFO o.a.s.f.p.FluxParser - Not performing property
substitution.
335 [main] INFO o.a.s.f.p.FluxParser - Not performing environment
variable substitution.
Exception in thread "main" java.lang.NoClassDefFoundError:
backtype/storm/spout/MultiScheme
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilder.java:350)
at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
at org.apache.storm.flux.Flux.runCli(Flux.java:153)
at org.apache.storm.flux.Flux.main(Flux.java:98)
Caused by: java.lang.ClassNotFoundException:
backtype.storm.spout.MultiScheme
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
This used to work fine with 0.10.0
I have checked all the places in my code including mvn dependency:tree's
output
All versions are 1.0.2 (storm and flux) and there is no mention of backtype
anywhere in the code, flux-file or the jar we are deploying.
What is wrong here?
Thanks
SG
Re: Flux exception while upgrading to Storm 1.0.2
Posted by S G <sg...@gmail.com>.
Hi Manu,
I do not see any 0.10.0 versions being downloaded in my ~/.m2 repo:
ls ~/.m2/repository/org/apache/storm/*
~/.m2/repository/org/apache/storm/flux:
1.0.2
~/.m2/repository/org/apache/storm/flux-core:
1.0.2
~/.m2/repository/org/apache/storm/storm:
1.0.2
~/.m2/repository/org/apache/storm/storm-kafka:
1.0.2
And yet surprisingly, when I do `mvn clean install` and explore the jar
file, I see two versions of KeyValueSchemeAsMultiScheme.class
jar -xvf my_shaded_fat_jar.jar
find . -name KeyValueSchemeAsMultiScheme.class
./org/apache/storm/kafka/KeyValueSchemeAsMultiScheme.class
./storm/kafka/KeyValueSchemeAsMultiScheme.class
On Mon, Sep 12, 2016 at 5:56 PM, Manu Zhang <ow...@gmail.com> wrote:
> Hi,
>
> Could you share your maven file ? When running `mvn package`, you should
> see what and which version of jar files are included ?
> That may give you a hint.
>
> Thanks,
> Manu
>
>
> On Tue, Sep 13, 2016 at 6:50 AM S G <sg...@gmail.com> wrote:
>
> > From the
> > https://github.com/apache/storm/tree/1.0.x-branch/external/storm-kafka
> > docs,
> >
> > "This means that pre 1.0 kafka spouts will not work with Storm versions
> 1.0
> > and higher. While running topologies in Storm version 1.0 and higher, it
> > must be ensured that the storm-kafka version is at least 1.0. Pre 1.0
> > shaded topology jars that bundles storm-kafka classes must be rebuilt
> with
> > storm-kafka version 1.0 for running in clusters with storm 1.0 and
> higher."
> >
> > Any idea how do I ensure that?
> > I have done a thorough checking of my pom.xmls and also ran grep
> > expressions.
> >
> > On Mon, Sep 12, 2016 at 3:44 PM, S G <sg...@gmail.com> wrote:
> >
> > > Can anyone help with this?
> > >
> > > I tried relocating classes too during shading as:
> > > <relocations>
> > > <relocation>
> > > <pattern>backtype.storm</pattern>
> > > <shadedPattern>org.apache.storm</shadedPattern>
> > > </relocation>
> > > <relocation>
> > > <pattern>storm.trident</pattern>
> > > <shadedPattern>org.apache.storm.trident</shadedPattern>
> > > </relocation>
> > > <relocation>
> > > <pattern>org.apache.thrift7</pattern>
> > > <shadedPattern>org.apache.storm.thrift</shadedPattern>
> > > </relocation>
> > > </relocations>
> > >
> > > But then the deployed topology began to complain about:
> > > java.lang.NoSuchMethodError: org.apache.storm.spout.
> > > MultiScheme.deserialize([B)Ljava/lang/Iterable; at
> > storm.kafka.KafkaUtils.
> > > generateTuples(KafkaUtils.java:206)
> > >
> > > So it seems that I really need to get the 1.0.2 storm-kafka into my
> > > project to make it work.
> > >
> > > I even tried to exclude storm-kafka everywhere from all the
> dependencies
> > > my project had and then including it as an explicit dependency, but it
> > > still did not work.
> > >
> > > mvn dependency:tree shows that storm-kafka is nowhere being included
> as a
> > > transitive dependency in my project.
> > >
> > > What to do now?
> > >
> > > Thx,
> > > SG
> > >
> > > On Mon, Sep 12, 2016 at 12:24 PM, S G <sg...@gmail.com>
> wrote:
> > >
> > >> Hi,
> > >>
> > >> We are upgrading to Storm 1.0.2 and have done all the package name
> > >> changes.
> > >>
> > >> grep -ri backtype in our code repo returns nothing.
> > >>
> > >> Yet on deploying the jar to Storm, following error is seen:
> > >>
> > >>
> > >> storm/storm_installed/apache-storm-1.0.2/bin/storm \
> > >> jar \
> > >> $jarFile \
> > >> org.apache.storm.flux.Flux \
> > >> $yamlFileTarget \
> > >> --remote
> > >>
> > >>
> > >>
> > >> +- Apache Storm -+
> > >> +- data FLow User eXperience -+
> > >> Version: 1.0.2
> > >> Parsing file: topology-config.yaml
> > >> 333 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input
> > stream...
> > >> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing property
> > >> substitution.
> > >> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing environment
> > >> variable substitution.
> > >> Exception in thread "main" java.lang.NoClassDefFoundError:
> > >> backtype/storm/spout/MultiScheme
> > >> at java.lang.Class.forName0(Native Method)
> > >> at java.lang.Class.forName(Class.java:264)
> > >> at org.apache.storm.flux.FluxBuilder.buildObject(
> FluxBuilder.java:291)
> > >> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilde
> > >> r.java:350)
> > >> at org.apache.storm.flux.FluxBuilder.buildTopology(
> FluxBuilder.java:75)
> > >> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
> > >> at org.apache.storm.flux.Flux.main(Flux.java:98)
> > >> Caused by: java.lang.ClassNotFoundException:
> > >> backtype.storm.spout.MultiScheme
> > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > >> ... 7 more
> > >>
> > >>
> > >>
> > >> This used to work fine with 0.10.0
> > >>
> > >> I have checked all the places in my code including mvn
> dependency:tree's
> > >> output
> > >> All versions are 1.0.2 (storm and flux) and there is no mention of
> > >> backtype anywhere in the code, flux-file or the jar we are deploying.
> > >>
> > >> What is wrong here?
> > >>
> > >> Thanks
> > >> SG
> > >>
> > >
> > >
> >
>
Re: Flux exception while upgrading to Storm 1.0.2
Posted by Manu Zhang <ow...@gmail.com>.
Hi,
Could you share your maven file ? When running `mvn package`, you should
see what and which version of jar files are included ?
That may give you a hint.
Thanks,
Manu
On Tue, Sep 13, 2016 at 6:50 AM S G <sg...@gmail.com> wrote:
> From the
> https://github.com/apache/storm/tree/1.0.x-branch/external/storm-kafka
> docs,
>
> "This means that pre 1.0 kafka spouts will not work with Storm versions 1.0
> and higher. While running topologies in Storm version 1.0 and higher, it
> must be ensured that the storm-kafka version is at least 1.0. Pre 1.0
> shaded topology jars that bundles storm-kafka classes must be rebuilt with
> storm-kafka version 1.0 for running in clusters with storm 1.0 and higher."
>
> Any idea how do I ensure that?
> I have done a thorough checking of my pom.xmls and also ran grep
> expressions.
>
> On Mon, Sep 12, 2016 at 3:44 PM, S G <sg...@gmail.com> wrote:
>
> > Can anyone help with this?
> >
> > I tried relocating classes too during shading as:
> > <relocations>
> > <relocation>
> > <pattern>backtype.storm</pattern>
> > <shadedPattern>org.apache.storm</shadedPattern>
> > </relocation>
> > <relocation>
> > <pattern>storm.trident</pattern>
> > <shadedPattern>org.apache.storm.trident</shadedPattern>
> > </relocation>
> > <relocation>
> > <pattern>org.apache.thrift7</pattern>
> > <shadedPattern>org.apache.storm.thrift</shadedPattern>
> > </relocation>
> > </relocations>
> >
> > But then the deployed topology began to complain about:
> > java.lang.NoSuchMethodError: org.apache.storm.spout.
> > MultiScheme.deserialize([B)Ljava/lang/Iterable; at
> storm.kafka.KafkaUtils.
> > generateTuples(KafkaUtils.java:206)
> >
> > So it seems that I really need to get the 1.0.2 storm-kafka into my
> > project to make it work.
> >
> > I even tried to exclude storm-kafka everywhere from all the dependencies
> > my project had and then including it as an explicit dependency, but it
> > still did not work.
> >
> > mvn dependency:tree shows that storm-kafka is nowhere being included as a
> > transitive dependency in my project.
> >
> > What to do now?
> >
> > Thx,
> > SG
> >
> > On Mon, Sep 12, 2016 at 12:24 PM, S G <sg...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> We are upgrading to Storm 1.0.2 and have done all the package name
> >> changes.
> >>
> >> grep -ri backtype in our code repo returns nothing.
> >>
> >> Yet on deploying the jar to Storm, following error is seen:
> >>
> >>
> >> storm/storm_installed/apache-storm-1.0.2/bin/storm \
> >> jar \
> >> $jarFile \
> >> org.apache.storm.flux.Flux \
> >> $yamlFileTarget \
> >> --remote
> >>
> >>
> >>
> >> +- Apache Storm -+
> >> +- data FLow User eXperience -+
> >> Version: 1.0.2
> >> Parsing file: topology-config.yaml
> >> 333 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input
> stream...
> >> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing property
> >> substitution.
> >> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing environment
> >> variable substitution.
> >> Exception in thread "main" java.lang.NoClassDefFoundError:
> >> backtype/storm/spout/MultiScheme
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:264)
> >> at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
> >> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilde
> >> r.java:350)
> >> at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
> >> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
> >> at org.apache.storm.flux.Flux.main(Flux.java:98)
> >> Caused by: java.lang.ClassNotFoundException:
> >> backtype.storm.spout.MultiScheme
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >> ... 7 more
> >>
> >>
> >>
> >> This used to work fine with 0.10.0
> >>
> >> I have checked all the places in my code including mvn dependency:tree's
> >> output
> >> All versions are 1.0.2 (storm and flux) and there is no mention of
> >> backtype anywhere in the code, flux-file or the jar we are deploying.
> >>
> >> What is wrong here?
> >>
> >> Thanks
> >> SG
> >>
> >
> >
>
Re: Flux exception while upgrading to Storm 1.0.2
Posted by S G <sg...@gmail.com>.
From the
https://github.com/apache/storm/tree/1.0.x-branch/external/storm-kafka docs,
"This means that pre 1.0 kafka spouts will not work with Storm versions 1.0
and higher. While running topologies in Storm version 1.0 and higher, it
must be ensured that the storm-kafka version is at least 1.0. Pre 1.0
shaded topology jars that bundles storm-kafka classes must be rebuilt with
storm-kafka version 1.0 for running in clusters with storm 1.0 and higher."
Any idea how do I ensure that?
I have done a thorough checking of my pom.xmls and also ran grep
expressions.
On Mon, Sep 12, 2016 at 3:44 PM, S G <sg...@gmail.com> wrote:
> Can anyone help with this?
>
> I tried relocating classes too during shading as:
> <relocations>
> <relocation>
> <pattern>backtype.storm</pattern>
> <shadedPattern>org.apache.storm</shadedPattern>
> </relocation>
> <relocation>
> <pattern>storm.trident</pattern>
> <shadedPattern>org.apache.storm.trident</shadedPattern>
> </relocation>
> <relocation>
> <pattern>org.apache.thrift7</pattern>
> <shadedPattern>org.apache.storm.thrift</shadedPattern>
> </relocation>
> </relocations>
>
> But then the deployed topology began to complain about:
> java.lang.NoSuchMethodError: org.apache.storm.spout.
> MultiScheme.deserialize([B)Ljava/lang/Iterable; at storm.kafka.KafkaUtils.
> generateTuples(KafkaUtils.java:206)
>
> So it seems that I really need to get the 1.0.2 storm-kafka into my
> project to make it work.
>
> I even tried to exclude storm-kafka everywhere from all the dependencies
> my project had and then including it as an explicit dependency, but it
> still did not work.
>
> mvn dependency:tree shows that storm-kafka is nowhere being included as a
> transitive dependency in my project.
>
> What to do now?
>
> Thx,
> SG
>
> On Mon, Sep 12, 2016 at 12:24 PM, S G <sg...@gmail.com> wrote:
>
>> Hi,
>>
>> We are upgrading to Storm 1.0.2 and have done all the package name
>> changes.
>>
>> grep -ri backtype in our code repo returns nothing.
>>
>> Yet on deploying the jar to Storm, following error is seen:
>>
>>
>> storm/storm_installed/apache-storm-1.0.2/bin/storm \
>> jar \
>> $jarFile \
>> org.apache.storm.flux.Flux \
>> $yamlFileTarget \
>> --remote
>>
>>
>>
>> +- Apache Storm -+
>> +- data FLow User eXperience -+
>> Version: 1.0.2
>> Parsing file: topology-config.yaml
>> 333 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
>> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing property
>> substitution.
>> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing environment
>> variable substitution.
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> backtype/storm/spout/MultiScheme
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:264)
>> at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
>> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilde
>> r.java:350)
>> at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
>> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
>> at org.apache.storm.flux.Flux.main(Flux.java:98)
>> Caused by: java.lang.ClassNotFoundException:
>> backtype.storm.spout.MultiScheme
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> ... 7 more
>>
>>
>>
>> This used to work fine with 0.10.0
>>
>> I have checked all the places in my code including mvn dependency:tree's
>> output
>> All versions are 1.0.2 (storm and flux) and there is no mention of
>> backtype anywhere in the code, flux-file or the jar we are deploying.
>>
>> What is wrong here?
>>
>> Thanks
>> SG
>>
>
>
Re: Flux exception while upgrading to Storm 1.0.2
Posted by S G <sg...@gmail.com>.
Can anyone help with this?
I tried relocating classes too during shading as:
<relocations>
<relocation>
<pattern>backtype.storm</pattern>
<shadedPattern>org.apache.storm</shadedPattern>
</relocation>
<relocation>
<pattern>storm.trident</pattern>
<shadedPattern>org.apache.storm.trident</shadedPattern>
</relocation>
<relocation>
<pattern>org.apache.thrift7</pattern>
<shadedPattern>org.apache.storm.thrift</shadedPattern>
</relocation>
</relocations>
But then the deployed topology began to complain about:
java.lang.NoSuchMethodError:
org.apache.storm.spout.MultiScheme.deserialize([B)Ljava/lang/Iterable; at
storm.kafka.KafkaUtils.generateTuples(KafkaUtils.java:206)
So it seems that I really need to get the 1.0.2 storm-kafka into my project
to make it work.
I even tried to exclude storm-kafka everywhere from all the dependencies my
project had and then including it as an explicit dependency, but it still
did not work.
mvn dependency:tree shows that storm-kafka is nowhere being included as a
transitive dependency in my project.
What to do now?
Thx,
SG
On Mon, Sep 12, 2016 at 12:24 PM, S G <sg...@gmail.com> wrote:
> Hi,
>
> We are upgrading to Storm 1.0.2 and have done all the package name changes.
>
> grep -ri backtype in our code repo returns nothing.
>
> Yet on deploying the jar to Storm, following error is seen:
>
>
> storm/storm_installed/apache-storm-1.0.2/bin/storm \
> jar \
> $jarFile \
> org.apache.storm.flux.Flux \
> $yamlFileTarget \
> --remote
>
>
>
> +- Apache Storm -+
> +- data FLow User eXperience -+
> Version: 1.0.2
> Parsing file: topology-config.yaml
> 333 [main] INFO o.a.s.f.p.FluxParser - loading YAML from input stream...
> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing property
> substitution.
> 335 [main] INFO o.a.s.f.p.FluxParser - Not performing environment
> variable substitution.
> Exception in thread "main" java.lang.NoClassDefFoundError:
> backtype/storm/spout/MultiScheme
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.apache.storm.flux.FluxBuilder.buildObject(FluxBuilder.java:291)
> at org.apache.storm.flux.FluxBuilder.buildComponents(FluxBuilder.java:350)
> at org.apache.storm.flux.FluxBuilder.buildTopology(FluxBuilder.java:75)
> at org.apache.storm.flux.Flux.runCli(Flux.java:153)
> at org.apache.storm.flux.Flux.main(Flux.java:98)
> Caused by: java.lang.ClassNotFoundException: backtype.storm.spout.
> MultiScheme
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 7 more
>
>
>
> This used to work fine with 0.10.0
>
> I have checked all the places in my code including mvn dependency:tree's
> output
> All versions are 1.0.2 (storm and flux) and there is no mention of
> backtype anywhere in the code, flux-file or the jar we are deploying.
>
> What is wrong here?
>
> Thanks
> SG
>