[jira] [Commented] (BEAM-1644) IO ITs: shared directory for kubernetes resources and PipelineOptions?

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927060#comment-15927060 ] 

ASF GitHub Bot commented on BEAM-1644:
--------------------------------------

GitHub user ssisk opened a pull request:

    https://github.com/apache/beam/pull/2253

    [BEAM-1644] Move PipelineOptions for IO ITs into shared location.

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [X] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [X] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [X] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
    
    ---
    * creates a common directory for IO 
    * moves the current pipeline options (which are spread out into separate directories) into that common directory.
    
    This is is useful/necessary because:
    1. In order to run all the IO ITs in one run, they have to have a shared set of pipeline options - options on the command line but not present in the PipelineOptions being used to read them will cause an error
    2. Data stores may be accessed in different IO modules, but should share common command line options - having them in a common directory makes that easier.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ssisk/beam io-test-pipeline-options

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/2253.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2253
    
----
commit c5568417876ed6c8ea2e64c6caa22098f4494d41
Author: Stephen Sisk <si...@google.com>
Date:   2017-03-15T21:31:49Z

    Move PipelineOptions for IO ITs into shared location.

----


> IO ITs: shared directory for kubernetes resources and PipelineOptions?
> ----------------------------------------------------------------------
>
>                 Key: BEAM-1644
>                 URL: https://issues.apache.org/jira/browse/BEAM-1644
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Stephen Sisk
>            Assignee: Stephen Sisk
>
> It is the case that different IOs will be created that connect to the same data stores - HadoopInputFormat in particular uses ES and cassandra, which are also used in their respective IOs as well. Jdbc is likely to have the same type of overlap.
> It would be nice to share TestPipelineOption and kubernetes/docker scripts so that we don't need to repeat them in each module. 
> As such, we should put them in a shared location. It seems like the best bet is to create a new module for test resources. 
> Proposal: 
> * Create a new module under io, test-common (? I'm not in love with that name) 
> * All the IOs with ITs can put their TestPipelineOption classes there
> * it will have a kubernetes directory with the various kubernetes scripts in sub directories
> Further thoughts:
> If 2 IOs need different configurations of a data store, this would make that harder. In that case, there's nothing stopping us from having two different sets of the kubernetes scripts in the shared location



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)