[jira] [Created] (BEAM-1807) IO ITs: shared language neutral directory for kubernetes resources

["Stephen Sisk (JIRA)" <ji...@apache.org>]

Stephen Sisk created BEAM-1807:
----------------------------------

             Summary: IO ITs: shared language neutral directory for kubernetes resources
                 Key: BEAM-1807
                 URL: https://issues.apache.org/jira/browse/BEAM-1807
             Project: Beam
          Issue Type: Improvement
          Components: testing
            Reporter: Stephen Sisk
            Assignee: Davor Bonaci


This is a follow-up to BEAM-1644. As was discussed there: 

"
It is the case that different IOs will be created that connect to the same data stores - HadoopInputFormat in particular uses ES and cassandra, which are also used in their respective IOs as well. Jdbc is likely to have the same type of overlap.

It would be nice to share [...] kubernetes/docker scripts so that we don't need to repeat them in each module. 
"

For BEAM-1644, we created a directory for java io-common resources - that's perfect for the java pipeline options we needed. However, we shouldn't put kubernetes resources in the newly created sdks/java/io/common because that'd indicate that the scripts are java specific. 

It's also worth noting that we have this problem already for jenkins and travis, and solved it by creating .jenkins and .travis directories at the top-level.

Proposal
=======
move .jenkins and .travis into a new top level ".test-infra" folder, and put a kubernetes directory there.

So the new structure would look like:

.test-infra
  jenkins
  travis
  kubernetes
sdks
runners
examples
...

I don't know if travis/jenkins must look in .travis/.jenkins directories or if those are things that we can change. If both do, would lessen my excitement, but if at least one other thing can share that directory, that would make it worthwhile in my mind.

Alternate proposal
===============
add a top-level .kubernetes directory alongside .jenkins/.travis. 

I'm not a huge fan of this since I'd love to not add more top level clutter.


Alternate proposal
===============
We could create:
sdks/common/test-infra/kubernetes
and put the scripts there. 

I don't like this option as much because it's kind of a just a random directory and is disconnected from the rest of the test infrastructure scripts that we use. I also prefer the other option since it reduces the amount of top-level clutter.

Thoughts?

cc [~jasonkuster] [~davor] [~iemejia]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)