You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Romain Manni-Bucau <rm...@gmail.com> on 2018/02/13 17:48:35 UTC

classloader fixes for pipeline options

Hi guys,

just wanted to send a head ups I'm trying to make the pipeline options
classloader usage being enhanced to support an execution where the
sdks-java-core is not in the same classloader than the runner. I sent a
functional PR on that aspect at https://github.com/apache/beam/pull/4674 -
side note: if it is ok to eagerly merge without test I'm fine doing another
PR later to add the test.

I'm planning to add tests later this week but would like to give this
"small" fix some more noise because I think we should try to be better at
handling classloaders and not let our code use the "defaulting pattern"
currentClass.getClassLoader() when we are not sure what to use.

To give a small bakground on where I hit that: I have a test framework
supporting to switch the runner depending some config:

@Environment(ContextualEnvironment.class)@Environment(DirectRunnerEnvironment.class)@Environment(FlinkRunnerEnvironment.class)@Environment(SparkRunnerEnvironment.class)class
TheTransformTest {
    @EnvironmentalTest
    void testWithStandaloneAndBeamEnvironments() {
        from("beam://in?config=xxxx")
            .to("beam://out")
            .create()
            .execute();
        // add asserts on the output if needed
    }
}


This will execute the method 4 times (once per env).

Thanks,
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>