[jira] [Comment Edited] (BEAM-436) DirectRunner RunnableOnService tempLocation configuration insufficient

["Scott Wegner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15684080#comment-15684080 ] 

Scott Wegner edited comment on BEAM-436 at 11/21/16 4:56 PM:
-------------------------------------------------------------

I spoke w/ [~kenn] on Friday, and he pointed out that once we have a runner-independent graph surgery API, these tests can transition from RunnableOnService to NeedsRunner, and we can initialize using a temp directory similar to other tests (TextIO NeedsRunner tests, for example).

Since the graph surgery work is underway, I propose to defer this work and later convert these tests to NeedsRunner. ([~tgroh] is there a JIRA I can link?)


was (Author: swegner):
I spoke w/ [~kenn] on Friday, and he pointed out that once we have a runner-independent graph surgery API, these tests can transition from RunnableOnService to NeedsRunner, and we can initialize using a temp directory similar to other tests (TextIO NeedsRunner tests, for example).

Since the graph surgery work is underway, I propose to defer this work and later convert these tests to NeedsRunner.

> DirectRunner RunnableOnService tempLocation configuration insufficient
> ----------------------------------------------------------------------
>
>                 Key: BEAM-436
>                 URL: https://issues.apache.org/jira/browse/BEAM-436
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-direct
>            Reporter: Scott Wegner
>            Assignee: Scott Wegner
>
> The way we specify temp storage locations for RunnableOnService tests is not sufficient, specifically for DirectRunner execution. Right now, RunnableOnService tests are run for DirectRunner and DataflowRunner, who set their temp locations differently:
> * DirectRunner doesn't specify a temp location directly, but test classes will use a JUnit @Rule TemporaryDirectory. Individual tests set it as necessary for tempLocation, and set a fake gs:// path for individual GCP IO tests.
> * DataflowRunner tests pass an actual GCS path as tempRoot, and TestDataflowRunner will initialize stagingLocation to this path.
> This setup makes it difficult to write RunnableOnService tests which pass for both runners. We should separate temp location setup out of individual test classes so that RunnableOnService tests "just work" on any runner.
> One solution would be to add logic inside TestPipeline#testingPipelineOptions:
> # If --tempRoot is specified, use it to set tempLocation and stagingLocation. Otherwise, use a JUnit TemporaryDirectory to set it
> # If tempLocation is a GCS path, use it to set stagingLocation. Otherwise, use a fake gcs path (i.e. gs://foo)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)