You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Yi Pan <ni...@gmail.com> on 2019/01/31 03:01:00 UTC
Example project configuration (maven or gradle) for projects
depending on BeamSQL sdk extensions
Hi, all,
Newbie here trying to figure out how to use published
beam-sdks-java-extensions-sql-2.9.0 in my own project.
I tried to create a gradle project to use BeamSQL sdk libraries. Here is
the build.gradle I have:
{code}
plugins {
id 'java'
}
group 'com.mycompany.myproject'
version '1.0-SNAPSHOT'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
apply plugin: 'java'
sourceSets.main.java.srcDirs = [
'src/main/java'
]
dependencies {
compile 'org.apache.beam:beam-sdks-java-core:2.9.0'
compile 'org.apache.beam:beam-sdks-java-extensions-sql:2.9.0'
compile 'com.google.code.findbugs:jsr305:3.0.2'
runtime 'org.apache.beam:beam-runners-direct-java:2.9.0'
testCompile group: 'junit', name: 'junit', version: '4.12'
}
// Run basic SQL example
task runBasicExample(type: JavaExec) {
description = "Run basic SQL example"
main = "com.mycompany.myproject.streamsql.examples.BeamSQLExample"
classpath = sourceSets.main.runtimeClasspath
args = ["--runner=DirectRunner"]
println classpath.getAsPath()
println args
}
{code}
The example BeamSQLExample is just copied from
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
.
I was able to compile the example in JDK8. However, when I tried to run it,
I hit the following exception:
{code}
Exception in thread "main" java.util.ServiceConfigurationError:
org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider:
Provider org.apache.beam.sdk.extensions.sql.impl.udf.BuiltinStringFunctions
could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.loadBeamBuiltinFunctions(BeamSqlEnv.java:128)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:94)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
at
com.mycompany.myprject.streamsql.examples.BeamSQLExample.main(BeamSQLExample.java:72)
Caused by: java.lang.NoClassDefFoundError:
org/apache/commons/codec/DecoderException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 9 more
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.codec.DecoderException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
{code}
When I traced into the code, it turns out that the failure occurs when the
classloader tries to get the default constructor w/o any parameters for
BuiltinStringFunctions.class. I double checked my local gradle cache and
confirmed that the jar is there:
{code}
SJCMAC91THJHD4:beamsql-demo ypan$ ls -l
~/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
-rw-r--r-- 1 ypan 192360288 12761025 Jan 29 18:02
/Users/ypan/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
{code}
I also have tried to compile the sql sdk libraries in Beam's source repo
and just copying over the generated class files over to my runtime
classpath. Apparently, that did not work well since there are many shadowed
libraries now need explicit declaring dependencies. I tried to search for
an example of maven or gradle dependency declaration that users will use to
create their own application using the SQL SDKs but was not lucky enough to
find one.
Given the amount of debugging and searching effort I have put it there, I
think that it is reasonable to ask the question to the community. Can
someone shed some light on what went wrong in my build project?
Thanks!
-Yi
Re: Example project configuration (maven or gradle) for projects
depending on BeamSQL sdk extensions
Posted by Yi Pan <ni...@gmail.com>.
Hi, Kenn,
Thanks for the quick response! Just FYI, I downgrade to 2.8.0 and the same
project runs fine now. Will update the ticket accordingly.
-Yi
On Wed, Jan 30, 2019 at 9:11 PM Kenneth Knowles <kl...@google.com> wrote:
> Wow, thanks for the great report. Your configuration looks good to me. I
> filed https://issues.apache.org/jira/browse/BEAM-6558 to figure this out.
>
> Kenn
>
> On Wed, Jan 30, 2019 at 7:01 PM Yi Pan <ni...@gmail.com> wrote:
>
>> Hi, all,
>>
>> Newbie here trying to figure out how to use published
>> beam-sdks-java-extensions-sql-2.9.0 in my own project.
>>
>> I tried to create a gradle project to use BeamSQL sdk libraries. Here is
>> the build.gradle I have:
>> {code}
>> plugins {
>> id 'java'
>> }
>>
>> group 'com.mycompany.myproject'
>> version '1.0-SNAPSHOT'
>>
>> sourceCompatibility = 1.8
>>
>> repositories {
>> mavenCentral()
>> }
>>
>> apply plugin: 'java'
>>
>> sourceSets.main.java.srcDirs = [
>> 'src/main/java'
>> ]
>>
>> dependencies {
>> compile 'org.apache.beam:beam-sdks-java-core:2.9.0'
>> compile 'org.apache.beam:beam-sdks-java-extensions-sql:2.9.0'
>> compile 'com.google.code.findbugs:jsr305:3.0.2'
>> runtime 'org.apache.beam:beam-runners-direct-java:2.9.0'
>> testCompile group: 'junit', name: 'junit', version: '4.12'
>> }
>>
>> // Run basic SQL example
>> task runBasicExample(type: JavaExec) {
>> description = "Run basic SQL example"
>> main = "com.mycompany.myproject.streamsql.examples.BeamSQLExample"
>> classpath = sourceSets.main.runtimeClasspath
>> args = ["--runner=DirectRunner"]
>> println classpath.getAsPath()
>> println args
>> }
>> {code}
>>
>> The example BeamSQLExample is just copied from
>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
>> .
>>
>> I was able to compile the example in JDK8. However, when I tried to run
>> it, I hit the following exception:
>> {code}
>> Exception in thread "main" java.util.ServiceConfigurationError:
>> org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider:
>> Provider org.apache.beam.sdk.extensions.sql.impl.udf.BuiltinStringFunctions
>> could not be instantiated
>> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
>> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
>> at
>> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
>> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>> at
>> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.loadBeamBuiltinFunctions(BeamSqlEnv.java:128)
>> at
>> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:94)
>> at
>> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76)
>> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
>> at
>> com.mycompany.myprject.streamsql.examples.BeamSQLExample.main(BeamSQLExample.java:72)
>> Caused by: java.lang.NoClassDefFoundError:
>> org/apache/commons/codec/DecoderException
>> at java.lang.Class.getDeclaredConstructors0(Native Method)
>> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
>> at java.lang.Class.getConstructor0(Class.java:3075)
>> at java.lang.Class.newInstance(Class.java:412)
>> at
>> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
>> ... 9 more
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.commons.codec.DecoderException
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> ... 14 more
>> {code}
>>
>> When I traced into the code, it turns out that the failure occurs when
>> the classloader tries to get the default constructor w/o any parameters for
>> BuiltinStringFunctions.class. I double checked my local gradle cache and
>> confirmed that the jar is there:
>> {code}
>> SJCMAC91THJHD4:beamsql-demo ypan$ ls -l
>> ~/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
>> -rw-r--r-- 1 ypan 192360288 12761025 Jan 29 18:02
>> /Users/ypan/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
>> {code}
>>
>> I also have tried to compile the sql sdk libraries in Beam's source repo
>> and just copying over the generated class files over to my runtime
>> classpath. Apparently, that did not work well since there are many shadowed
>> libraries now need explicit declaring dependencies. I tried to search for
>> an example of maven or gradle dependency declaration that users will use to
>> create their own application using the SQL SDKs but was not lucky enough to
>> find one.
>>
>> Given the amount of debugging and searching effort I have put it there, I
>> think that it is reasonable to ask the question to the community. Can
>> someone shed some light on what went wrong in my build project?
>>
>> Thanks!
>>
>> -Yi
>>
>>
Re: Example project configuration (maven or gradle) for projects
depending on BeamSQL sdk extensions
Posted by Kenneth Knowles <kl...@google.com>.
Wow, thanks for the great report. Your configuration looks good to me. I
filed https://issues.apache.org/jira/browse/BEAM-6558 to figure this out.
Kenn
On Wed, Jan 30, 2019 at 7:01 PM Yi Pan <ni...@gmail.com> wrote:
> Hi, all,
>
> Newbie here trying to figure out how to use published
> beam-sdks-java-extensions-sql-2.9.0 in my own project.
>
> I tried to create a gradle project to use BeamSQL sdk libraries. Here is
> the build.gradle I have:
> {code}
> plugins {
> id 'java'
> }
>
> group 'com.mycompany.myproject'
> version '1.0-SNAPSHOT'
>
> sourceCompatibility = 1.8
>
> repositories {
> mavenCentral()
> }
>
> apply plugin: 'java'
>
> sourceSets.main.java.srcDirs = [
> 'src/main/java'
> ]
>
> dependencies {
> compile 'org.apache.beam:beam-sdks-java-core:2.9.0'
> compile 'org.apache.beam:beam-sdks-java-extensions-sql:2.9.0'
> compile 'com.google.code.findbugs:jsr305:3.0.2'
> runtime 'org.apache.beam:beam-runners-direct-java:2.9.0'
> testCompile group: 'junit', name: 'junit', version: '4.12'
> }
>
> // Run basic SQL example
> task runBasicExample(type: JavaExec) {
> description = "Run basic SQL example"
> main = "com.mycompany.myproject.streamsql.examples.BeamSQLExample"
> classpath = sourceSets.main.runtimeClasspath
> args = ["--runner=DirectRunner"]
> println classpath.getAsPath()
> println args
> }
> {code}
>
> The example BeamSQLExample is just copied from
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/example/BeamSqlExample.java
> .
>
> I was able to compile the example in JDK8. However, when I tried to run
> it, I hit the following exception:
> {code}
> Exception in thread "main" java.util.ServiceConfigurationError:
> org.apache.beam.sdk.extensions.sql.impl.udf.BeamBuiltinFunctionProvider:
> Provider org.apache.beam.sdk.extensions.sql.impl.udf.BuiltinStringFunctions
> could not be instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.loadBeamBuiltinFunctions(BeamSqlEnv.java:128)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:94)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:76)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:357)
> at
> com.mycompany.myprject.streamsql.examples.BeamSQLExample.main(BeamSQLExample.java:72)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/commons/codec/DecoderException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.newInstance(Class.java:412)
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
> ... 9 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.codec.DecoderException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 14 more
> {code}
>
> When I traced into the code, it turns out that the failure occurs when the
> classloader tries to get the default constructor w/o any parameters for
> BuiltinStringFunctions.class. I double checked my local gradle cache and
> confirmed that the jar is there:
> {code}
> SJCMAC91THJHD4:beamsql-demo ypan$ ls -l
> ~/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
> -rw-r--r-- 1 ypan 192360288 12761025 Jan 29 18:02
> /Users/ypan/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-sdks-java-extensions-sql/2.9.0/67e7675519859ff332619c4c6ea5d26a505dbd50/beam-sdks-java-extensions-sql-2.9.0.jar
> {code}
>
> I also have tried to compile the sql sdk libraries in Beam's source repo
> and just copying over the generated class files over to my runtime
> classpath. Apparently, that did not work well since there are many shadowed
> libraries now need explicit declaring dependencies. I tried to search for
> an example of maven or gradle dependency declaration that users will use to
> create their own application using the SQL SDKs but was not lucky enough to
> find one.
>
> Given the amount of debugging and searching effort I have put it there, I
> think that it is reasonable to ask the question to the community. Can
> someone shed some light on what went wrong in my build project?
>
> Thanks!
>
> -Yi
>
>