Flux Issues launching a FluxShellSpout on Windows using a relative path

[Mauro Giusti <ma...@microsoft.com>]

Working on a Storm project using C# for Bolts and Spouts -  as I have a dependency on .Net/Mono for I/O -

We are using Flux for the topologies and FluxShellSpout to invoke a .net 4.5 executable that uses the storm net adapter (https://github.com/ziyunhx/storm-net-adapter)

It works when:

-          We invoke it via mono

-          We invoke it natively using the full path of the exe

It does not work when we invoke it natively with a relative path.

We built a .net 4.5 exe using Sotrm.Net.Adapter and binplace it in a resource path:
<basedir>/multilang/resources/mono/RdfeExtractSpout.exe

The pom includes this resource like this:
      <resource>
          <directory>${basedir}/multilang</directory>
      </resource>

We can see that the mono directory is correctly copied into the target\classes folder when we compile the pom.

This topology works:
spouts:                 # Spout definitions
- id: "availability-csharp-spout"
   className: "org.apache.storm.flux.wrappers.spouts.FluxShellSpout"
   constructorArgs:
     # command line
     - ["mono", "mono/RdfeExtractSpout.exe"]
     # output fields
     - ["Service", "EnvironmentType", "EnvironmentName", "Plane", "TimeEmitted1MinBin", "GeoLocale", "Operation", "MetricLatency"]
   parallelism: 1


The spout runs under mono correctly on the Windows box.

This topology works too:
spouts:                 # Spout definitions
- id: "availability-csharp-spout"
   className: "org.apache.storm.flux.wrappers.spouts.FluxShellSpout"
   constructorArgs:
     # command line
     - ["C:\\OneBranch\\SRE\\DataInsightsHub\\src\\StormDataHub\\Topologies\\Rdfe\\target\\classes\\resources\\mono\\RdfeExtractSpout.exe"]
     # output fields
     - ["Service", "EnvironmentType", "EnvironmentName", "Plane", "TimeEmitted1MinBin", "GeoLocale", "Operation", "MetricLatency"]
   parallelism: 1

The spout runs under .net on the windows box (I know this would never work in production, but I am trying to make a point with the following config).

This topology does not work:
spouts:                 # Spout definitions
- id: "availability-csharp-spout"
   className: "org.apache.storm.flux.wrappers.spouts.FluxShellSpout"
   constructorArgs:
     # command line
     - ["mono\\RdfeExtractSpout.exe"]
     # output fields
     - ["Service", "EnvironmentType", "EnvironmentName", "Plane", "TimeEmitted1MinBin", "GeoLocale", "Operation", "MetricLatency"]
   parallelism: 1

I get the following:

16:52:50 [Thread-18-availability-csharp-spout-executor[2 2]] ERROR org.apache.storm.util - Async loop died!
java.lang.RuntimeException: Error when launching multilang subprocess

        at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:94) ~[storm-core-1.1.0.jar:1.1.0]
        at org.apache.storm.spout.ShellSpout.open(ShellSpout.java:111) ~[storm-core-1.1.0.jar:1.1.0]
        at org.apache.storm.daemon.executor$fn__4976$fn__4991.invoke(executor.clj:600) ~[storm-core-1.1.0.jar:1.1.0]
        at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.1.0.jar:1.1.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.io.IOException: Cannot run program "mono\RdfeExtractSpout.exe" (in directory "C:\Users\maurgi\AppData\Local\Temp\85a06845-1257-433b-a938-8ebe34
e6a43a\supervisor\stormdist\rdfe-sample-topology-1-1491954766\resources"): CreateProcess error=2, The system cannot find the file specified
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_121]
        at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:87) ~[storm-core-1.1.0.jar:1.1.0]


I tried different variants: mono\RdfeExtractSpout.exe , mono/RdfeExtractSpout.exe,
I tried to get rid of the mono subfolder and just call RdfeExtractSpout.exe but I get the same error.

The file is there in Windows, why can't the FluxShellSpout work with the relative path, but it can work with the absolute path?

[cid:image001.png@01D2B2E4.DD9828F0]

Thanks,
Mauro Giusti.