runShadow: prebuild and build in read-only directory

[Robert Lugg <Ro...@synopsys.com>]

I am able to run Beam through the python sdk using runShadow for the Flink runner.  I manually start runShadow using:

./gradlew -g ~/.gradle :runners:flink:1.8:job-server:runShadow

The beam directory, by necessity is in a read-only file system.  The only way I can get this to work is:

  1.  Copy ./beam/... to /tmp
  2.  Call gradlew from /tmp/beam

I would like to do a couple of things:
1) I would like to precompile runShadow such that it doesn't take a long time to run the first time it is called.  Is there any final binary, for instance, that I can just run, or must I always gradle build?
2) I would like to be able to run runShadow without needing to copy ./beam to a writable directory(ie /tmp).

Are these, or alternatives possible?

Here is the log indicating what happens if I try to build in the read-only directory.  Note that I did use the -g option:
```
./gradlew -g ~/.gradle :runners:flink:1.8:job-server:runShadow --info
Initialized native services in: /u/me/.gradle/native
The client will now receive all logging from the daemon (pid: 39022). The daemon log file: /u/me/.gradle/daemon/5.2.1/daemon-39022.out.log
Starting 4th build in daemon [uptime: 41.071 secs, performance: 100%, no major garbage collections]
Using 44 worker leases.

FAILURE: Build failed with an exception.

* What went wrong:
Could not create service of type ScriptPluginFactory using BuildScopeServices.createScriptPluginFactory().
> Could not create service of type CrossBuildFileHashCache using BuildSessionScopeServices.createCrossBuildFileHashCache().

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 0s
```
I apologize if I double posted.

RE: runShadow: prebuild and build in read-only directory

[Robert Lugg <Ro...@synopsys.com>]

Confirmed.  That worked perfectly.  Thank you.

Re: runShadow: prebuild and build in read-only directory

[Kyle Weaver <kc...@google.com>]

You can build the job server jar using:

./gradlew runners:flink:1.8:job-server:shadowJar

The output jar will be located in:

runners/flink/1.8/job-server/build/libs/

You can run the jar using `java -jar`.

Hope that helps.

On Thu, Jan 9, 2020 at 10:47 AM Robert Lugg <Ro...@synopsys.com>
wrote:

> I am able to run Beam through the python sdk using runShadow for the Flink
> runner.  I manually start runShadow using:
>
>
>
> ./gradlew -g ~/.gradle :runners:flink:1.8:job-server:runShadow
>
>
>
> The beam directory, by necessity is in a read-only file system.  The only
> way I can get this to work is:
>
>    1. Copy ./beam/… to /tmp
>    2. Call gradlew from /tmp/beam
>
>
>
> I would like to do a couple of things:
> 1) I would like to precompile runShadow such that it doesn’t take a long
> time to run the first time it is called.  Is there any final binary, for
> instance, that I can just run, or must I always gradle build?
>
> 2) I would like to be able to run runShadow without needing to copy ./beam
> to a writable directory(ie /tmp).
>
>
>
> Are these, or alternatives possible?
>
>
>
> Here is the log indicating what happens if I try to build in the read-only
> directory.  Note that I did use the -g option:
>
> ```
>
> ./gradlew -g ~/.gradle :runners:flink:1.8:job-server:runShadow --info
>
> Initialized native services in: /u/me/.gradle/native
>
> The client will now receive all logging from the daemon (pid: 39022). The
> daemon log file: /u/me/.gradle/daemon/5.2.1/daemon-39022.out.log
>
> Starting 4th build in daemon [uptime: 41.071 secs, performance: 100%, no
> major garbage collections]
>
> Using 44 worker leases.
>
>
>
> FAILURE: Build failed with an exception.
>
>
>
> * What went wrong:
>
> Could not create service of type ScriptPluginFactory using
> BuildScopeServices.createScriptPluginFactory().
>
> > Could not create service of type CrossBuildFileHashCache using
> BuildSessionScopeServices.createCrossBuildFileHashCache().
>
>
>
> * Try:
>
> Run with --stacktrace option to get the stack trace. Run with --debug
> option to get more log output. Run with --scan to get full insights.
>
>
>
> * Get more help at https://help.gradle.org
>
>
>
> BUILD FAILED in 0s
>
> ```
>
> I apologize if I double posted.
>