You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Nick R. Katsipoulakis" <ni...@gmail.com> on 2015/02/18 02:29:59 UTC
Found multiple defaults.yaml resources
Hello,
I am currently trying to automate some experiments on Storm and to do so, I
have created a Java process which in turn launches three Java sub-processes
using the ProcessBuilder framework. The three processes are mainly
executions of jar files (2 of them simple java applications, 1 of them
Storm topology submitted to a Storm cluster). Therefore, the 2 simple java
applications are executed as terminal commands:
$>java -jar executable-jar <arg-1> <arg-2> ... <arg-N>
The storm topology is bundled into a jar using Eclipse maven package
assembly:single goal and the following plugin configuration is added to the
pom.xml of the project:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.3</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The problem is that when I create the Java sub-process that submits the
following command:
$>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M>
at some point I get the following error:
Exception in thread "main" java.lang.RuntimeException: Found multiple
defaults.yaml resources. You're probably bundling the Storm jars with your
topology jar.
[jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml,
jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml]
at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140)
at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:191)
at backtype.storm.config$read_storm_config.invoke(config.clj:121)
at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
at clojure.lang.AFn.applyToHelper(AFn.java:161)
at clojure.lang.AFn.applyTo(AFn.java:151)
at backtype.storm.command.config_value.main(Unknown Source)
Therefore, I understand that during packaging, maven includes a
default.yaml file in my package, that conflicts with the yam file of the
cluster.
My question is how can I remove the default.yaml from my bundled-topology
or how can I overcome this issue. The simple solution would be to start my
topology by itself and then run the other two processes, but I want my
experiments to be fully automated.
Thank you and I apologize in advance for the long post.
Cheers,
Nikos
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Found multiple defaults.yaml resources
Posted by er...@gmail.com.
Hi Tammy!
For the I-944 form, should I prepare doc just for my wife or for both of us? (Like credit score, etc)
Thanks!
Erben
> On Feb 17, 2015, at 10:12 PM, Andrew Xor <an...@gmail.com> wrote:
>
>
> I concur to this; as a rule of the thumb I use compile when I develop locally (in LocalCluster) and provided if I deploy the topology to an actual cluster.
>
> Regards.
>
>> On Wed, Feb 18, 2015 at 7:23 AM, José Barrueta <jo...@stormpath.com> wrote:
>> Hey Nikon, I had the same issue days ago, the problem in my case was that I wasn't declaring the storm-core dependency as provided in the pom file, therefore the fat jar included the dependency, which is provided by nimbus (and workers) when deploying in the cluster.
>>
>> ...
>> <dependency>
>> <groupId>org.apache.storm</groupId>
>> <artifactId>storm-core</artifactId>
>> <version>${storm.version}</version>
>> <scope>provided</scope>
>> </dependency>
>> ...
>>
>> Best,
>>
>> Jose Luis
>>
>>> On Tue, Feb 17, 2015 at 5:29 PM, Nick R. Katsipoulakis <ni...@gmail.com> wrote:
>>> Hello,
>>>
>>> I am currently trying to automate some experiments on Storm and to do so, I have created a Java process which in turn launches three Java sub-processes using the ProcessBuilder framework. The three processes are mainly executions of jar files (2 of them simple java applications, 1 of them Storm topology submitted to a Storm cluster). Therefore, the 2 simple java applications are executed as terminal commands:
>>>
>>> $>java -jar executable-jar <arg-1> <arg-2> ... <arg-N>
>>>
>>> The storm topology is bundled into a jar using Eclipse maven package assembly:single goal and the following plugin configuration is added to the pom.xml of the project:
>>>
>>> <build>
>>> <plugins>
>>> <plugin>
>>> <groupId>org.apache.maven.plugins</groupId>
>>> <artifactId>maven-assembly-plugin</artifactId>
>>> <version>2.5.3</version>
>>> <configuration>
>>> <descriptorRefs>
>>> <descriptorRef>jar-with-dependencies</descriptorRef>
>>> </descriptorRefs>
>>> <archive>
>>> <manifest>
>>> <mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass>
>>> </manifest>
>>> </archive>
>>> </configuration>
>>> <executions>
>>> <execution>
>>> <phase>package</phase>
>>> <goals>
>>> <goal>single</goal>
>>> </goals>
>>> </execution>
>>> </executions>
>>> </plugin>
>>> </plugins>
>>> </build>
>>>
>>> The problem is that when I create the Java sub-process that submits the following command:
>>>
>>> $>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M>
>>>
>>> at some point I get the following error:
>>>
>>> Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml, jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml]
>>> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140)
>>> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167)
>>> at backtype.storm.utils.Utils.readStormConfig(Utils.java:191)
>>> at backtype.storm.config$read_storm_config.invoke(config.clj:121)
>>> at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
>>> at clojure.lang.AFn.applyToHelper(AFn.java:161)
>>> at clojure.lang.AFn.applyTo(AFn.java:151)
>>> at backtype.storm.command.config_value.main(Unknown Source)
>>>
>>> Therefore, I understand that during packaging, maven includes a default.yaml file in my package, that conflicts with the yam file of the cluster.
>>>
>>> My question is how can I remove the default.yaml from my bundled-topology or how can I overcome this issue. The simple solution would be to start my topology by itself and then run the other two processes, but I want my experiments to be fully automated.
>>>
>>> Thank you and I apologize in advance for the long post.
>>>
>>> Cheers,
>>> Nikos
>>>
>>> --
>>> Nikolaos Romanos Katsipoulakis,
>>> University of Pittsburgh, PhD candidate
>>
>
Re: Found multiple defaults.yaml resources
Posted by "Nick R. Katsipoulakis" <ni...@gmail.com>.
Hello,
Jose thank you for your useful response. I guess that is a better approach
and works.
Thanks,
Nikos
2015-02-18 1:11 GMT-05:00 Andrew Xor <an...@gmail.com>:
> I concur to this; as a rule of the thumb I use compile when I develop
> locally (in LocalCluster) and provided if I deploy the topology to an
> actual cluster.
>
> Regards.
>
> On Wed, Feb 18, 2015 at 7:23 AM, José Barrueta <jo...@stormpath.com> wrote:
>
>> Hey Nikon, I had the same issue days ago, the problem in my case was that
>> I wasn't declaring the storm-core dependency as provided in the pom file,
>> therefore the fat jar included the dependency, which is provided by nimbus
>> (and workers) when deploying in the cluster.
>>
>> ...
>> <dependency>
>> <groupId>org.apache.storm</groupId>
>> <artifactId>storm-core</artifactId>
>> <version>${storm.version}</version>
>> <scope>provided</scope>
>> </dependency>
>> ...
>>
>> Best,
>>
>> Jose Luis
>>
>> On Tue, Feb 17, 2015 at 5:29 PM, Nick R. Katsipoulakis <
>> nick.katsip@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am currently trying to automate some experiments on Storm and to do
>>> so, I have created a Java process which in turn launches three Java
>>> sub-processes using the ProcessBuilder framework. The three processes are
>>> mainly executions of jar files (2 of them simple java applications, 1 of
>>> them Storm topology submitted to a Storm cluster). Therefore, the 2 simple
>>> java applications are executed as terminal commands:
>>>
>>> $>java -jar executable-jar <arg-1> <arg-2> ... <arg-N>
>>>
>>> The storm topology is bundled into a jar using Eclipse maven package
>>> assembly:single goal and the following plugin configuration is added to the
>>> pom.xml of the project:
>>>
>>> <build>
>>>
>>> <plugins>
>>>
>>> <plugin>
>>>
>>> <groupId>org.apache.maven.plugins</groupId>
>>>
>>> <artifactId>maven-assembly-plugin</artifactId>
>>>
>>> <version>2.5.3</version>
>>>
>>> <configuration>
>>>
>>> <descriptorRefs>
>>>
>>> <descriptorRef>jar-with-dependencies</descriptorRef>
>>>
>>> </descriptorRefs>
>>>
>>> <archive>
>>>
>>> <manifest>
>>>
>>> <mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass>
>>>
>>> </manifest>
>>>
>>> </archive>
>>>
>>> </configuration>
>>>
>>> <executions>
>>>
>>> <execution>
>>>
>>> <phase>package</phase>
>>>
>>> <goals>
>>>
>>> <goal>single</goal>
>>>
>>> </goals>
>>>
>>> </execution>
>>>
>>> </executions>
>>>
>>> </plugin>
>>>
>>> </plugins>
>>>
>>> </build>
>>>
>>>
>>> The problem is that when I create the Java sub-process that submits the
>>> following command:
>>>
>>>
>>> $>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M>
>>>
>>>
>>> at some point I get the following error:
>>>
>>>
>>> Exception in thread "main" java.lang.RuntimeException: Found multiple
>>> defaults.yaml resources. You're probably bundling the Storm jars with your
>>> topology jar.
>>> [jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml,
>>> jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml]
>>>
>>> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140)
>>>
>>> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167)
>>>
>>> at backtype.storm.utils.Utils.readStormConfig(Utils.java:191)
>>>
>>> at backtype.storm.config$read_storm_config.invoke(config.clj:121)
>>>
>>> at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
>>>
>>> at clojure.lang.AFn.applyToHelper(AFn.java:161)
>>>
>>> at clojure.lang.AFn.applyTo(AFn.java:151)
>>>
>>> at backtype.storm.command.config_value.main(Unknown Source)
>>>
>>> Therefore, I understand that during packaging, maven includes a
>>> default.yaml file in my package, that conflicts with the yam file of the
>>> cluster.
>>>
>>> My question is how can I remove the default.yaml from my
>>> bundled-topology or how can I overcome this issue. The simple solution
>>> would be to start my topology by itself and then run the other two
>>> processes, but I want my experiments to be fully automated.
>>>
>>> Thank you and I apologize in advance for the long post.
>>>
>>> Cheers,
>>> Nikos
>>>
>>> --
>>> Nikolaos Romanos Katsipoulakis,
>>> University of Pittsburgh, PhD candidate
>>>
>>
>>
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Found multiple defaults.yaml resources
Posted by Andrew Xor <an...@gmail.com>.
I concur to this; as a rule of the thumb I use compile when I develop
locally (in LocalCluster) and provided if I deploy the topology to an
actual cluster.
Regards.
On Wed, Feb 18, 2015 at 7:23 AM, José Barrueta <jo...@stormpath.com> wrote:
> Hey Nikon, I had the same issue days ago, the problem in my case was that
> I wasn't declaring the storm-core dependency as provided in the pom file,
> therefore the fat jar included the dependency, which is provided by nimbus
> (and workers) when deploying in the cluster.
>
> ...
> <dependency>
> <groupId>org.apache.storm</groupId>
> <artifactId>storm-core</artifactId>
> <version>${storm.version}</version>
> <scope>provided</scope>
> </dependency>
> ...
>
> Best,
>
> Jose Luis
>
> On Tue, Feb 17, 2015 at 5:29 PM, Nick R. Katsipoulakis <
> nick.katsip@gmail.com> wrote:
>
>> Hello,
>>
>> I am currently trying to automate some experiments on Storm and to do so,
>> I have created a Java process which in turn launches three Java
>> sub-processes using the ProcessBuilder framework. The three processes are
>> mainly executions of jar files (2 of them simple java applications, 1 of
>> them Storm topology submitted to a Storm cluster). Therefore, the 2 simple
>> java applications are executed as terminal commands:
>>
>> $>java -jar executable-jar <arg-1> <arg-2> ... <arg-N>
>>
>> The storm topology is bundled into a jar using Eclipse maven package
>> assembly:single goal and the following plugin configuration is added to the
>> pom.xml of the project:
>>
>> <build>
>>
>> <plugins>
>>
>> <plugin>
>>
>> <groupId>org.apache.maven.plugins</groupId>
>>
>> <artifactId>maven-assembly-plugin</artifactId>
>>
>> <version>2.5.3</version>
>>
>> <configuration>
>>
>> <descriptorRefs>
>>
>> <descriptorRef>jar-with-dependencies</descriptorRef>
>>
>> </descriptorRefs>
>>
>> <archive>
>>
>> <manifest>
>>
>> <mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass>
>>
>> </manifest>
>>
>> </archive>
>>
>> </configuration>
>>
>> <executions>
>>
>> <execution>
>>
>> <phase>package</phase>
>>
>> <goals>
>>
>> <goal>single</goal>
>>
>> </goals>
>>
>> </execution>
>>
>> </executions>
>>
>> </plugin>
>>
>> </plugins>
>>
>> </build>
>>
>>
>> The problem is that when I create the Java sub-process that submits the
>> following command:
>>
>>
>> $>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M>
>>
>>
>> at some point I get the following error:
>>
>>
>> Exception in thread "main" java.lang.RuntimeException: Found multiple
>> defaults.yaml resources. You're probably bundling the Storm jars with your
>> topology jar.
>> [jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml,
>> jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml]
>>
>> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140)
>>
>> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167)
>>
>> at backtype.storm.utils.Utils.readStormConfig(Utils.java:191)
>>
>> at backtype.storm.config$read_storm_config.invoke(config.clj:121)
>>
>> at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
>>
>> at clojure.lang.AFn.applyToHelper(AFn.java:161)
>>
>> at clojure.lang.AFn.applyTo(AFn.java:151)
>>
>> at backtype.storm.command.config_value.main(Unknown Source)
>>
>> Therefore, I understand that during packaging, maven includes a
>> default.yaml file in my package, that conflicts with the yam file of the
>> cluster.
>>
>> My question is how can I remove the default.yaml from my bundled-topology
>> or how can I overcome this issue. The simple solution would be to start my
>> topology by itself and then run the other two processes, but I want my
>> experiments to be fully automated.
>>
>> Thank you and I apologize in advance for the long post.
>>
>> Cheers,
>> Nikos
>>
>> --
>> Nikolaos Romanos Katsipoulakis,
>> University of Pittsburgh, PhD candidate
>>
>
>
Re: Found multiple defaults.yaml resources
Posted by José Barrueta <jo...@stormpath.com>.
Hey Nikon, I had the same issue days ago, the problem in my case was that I
wasn't declaring the storm-core dependency as provided in the pom file,
therefore the fat jar included the dependency, which is provided by nimbus
(and workers) when deploying in the cluster.
...
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>${storm.version}</version>
<scope>provided</scope>
</dependency>
...
Best,
Jose Luis
On Tue, Feb 17, 2015 at 5:29 PM, Nick R. Katsipoulakis <
nick.katsip@gmail.com> wrote:
> Hello,
>
> I am currently trying to automate some experiments on Storm and to do so,
> I have created a Java process which in turn launches three Java
> sub-processes using the ProcessBuilder framework. The three processes are
> mainly executions of jar files (2 of them simple java applications, 1 of
> them Storm topology submitted to a Storm cluster). Therefore, the 2 simple
> java applications are executed as terminal commands:
>
> $>java -jar executable-jar <arg-1> <arg-2> ... <arg-N>
>
> The storm topology is bundled into a jar using Eclipse maven package
> assembly:single goal and the following plugin configuration is added to the
> pom.xml of the project:
>
> <build>
>
> <plugins>
>
> <plugin>
>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-assembly-plugin</artifactId>
>
> <version>2.5.3</version>
>
> <configuration>
>
> <descriptorRefs>
>
> <descriptorRef>jar-with-dependencies</descriptorRef>
>
> </descriptorRefs>
>
> <archive>
>
> <manifest>
>
> <mainClass>gr.katsip.storm.topology.ExperimentalTopology</mainClass>
>
> </manifest>
>
> </archive>
>
> </configuration>
>
> <executions>
>
> <execution>
>
> <phase>package</phase>
>
> <goals>
>
> <goal>single</goal>
>
> </goals>
>
> </execution>
>
> </executions>
>
> </plugin>
>
> </plugins>
>
> </build>
>
>
> The problem is that when I create the Java sub-process that submits the
> following command:
>
>
> $>storm jar bundled-topology.jar <arg-1> <arg-2> ... <arg-M>
>
>
> at some point I get the following error:
>
>
> Exception in thread "main" java.lang.RuntimeException: Found multiple
> defaults.yaml resources. You're probably bundling the Storm jars with your
> topology jar.
> [jar:file:/Users/nick/Documents/stream-gen/Experiment/synefo-topology.jar!/defaults.yaml,
> jar:file:/Users/nick/apache-storm/lib/storm-core-0.9.2-incubating.jar!/defaults.yaml]
>
> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:140)
>
> at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:167)
>
> at backtype.storm.utils.Utils.readStormConfig(Utils.java:191)
>
> at backtype.storm.config$read_storm_config.invoke(config.clj:121)
>
> at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
>
> at clojure.lang.AFn.applyToHelper(AFn.java:161)
>
> at clojure.lang.AFn.applyTo(AFn.java:151)
>
> at backtype.storm.command.config_value.main(Unknown Source)
>
> Therefore, I understand that during packaging, maven includes a
> default.yaml file in my package, that conflicts with the yam file of the
> cluster.
>
> My question is how can I remove the default.yaml from my bundled-topology
> or how can I overcome this issue. The simple solution would be to start my
> topology by itself and then run the other two processes, but I want my
> experiments to be fully automated.
>
> Thank you and I apologize in advance for the long post.
>
> Cheers,
> Nikos
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>