You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by Vicky Kak <vi...@gmail.com> on 2017/08/09 12:51:59 UTC
Gobbin build failure when using custom flavor
Hi All,
I have been building the tag gobblin_0.9.0 with kafka 9, here are the
changes that I did
1)
vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin$ git diff
diff --git a/gobblin-distribution/gobblin-flavor-custom.gradle
b/gobblin-distribution/gobblin-flavor-custom.gradle
index 5b363bd45..a890c4944 100644
--- a/gobblin-distribution/gobblin-flavor-custom.gradle
+++ b/gobblin-distribution/gobblin-flavor-custom.gradle
@@ -1,6 +1,6 @@
dependencies {
// Example jobs
// compile project(':gobblin-example')
-
+ compile project(':gobblin-modules:gobblin-kafka-09')
}
diff --git
a/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep
b/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep
deleted file mode 100644
index e69de29bb..000000000
diff --git
a/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep
b/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep
deleted file mode 100644
index e69de29bb..000000000
diff --git
a/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep
b/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep
deleted file mode 100644
index e69de29bb..000000000
diff --git a/settings.gradle b/settings.gradle
index c63463e0f..ac8db6940 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -41,7 +41,7 @@ def modules = ['gobblin-admin',
// Disable jacoco for now as Kafka 0.8 is the default version and jacoco
does not like the same classes
// being declared in different modules
def jacocoBlacklist = new HashSet([
- "gobblin-modules:gobblin-kafka-09"
+ "gobblin-modules:gobblin-kafka-08"
])
modules.each { module ->
2) I also added the gobblin-flavor-custom.gradle in the root directory
which is vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin
Finally I called
./gradlew -PgobblinFlavor=custom assemble
It created the gobblin distribution with the Kakfka0.9.
When I run the build which includes the test I got these issues
1) gobblin.metastore.DatabaseJobHistoryStoreV100Test:setUp failed with
following error
de.flapdoodle.embed.process.exceptions.DistributionException:
java.io.IOException: File
/home/vicky/.embedmysql/MySQL-5.6/mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz
at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:69)
at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.java:49)
at com.wix.mysql.EmbeddedMysql.<init>(EmbeddedMysql.java:39)
at com.wix.mysql.EmbeddedMysql$Builder.start(EmbeddedMysql.java:131)
at
gobblin.metastore.testing.TestMetastoreDatabaseServer.<init>(TestMetastoreDatabaseServer.java:94)
at
gobblin.metastore.testing.TestMetastoreDatabaseFactory.ensureDatabaseExists(TestMetastoreDatabaseFactory.java:72)
at
gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(TestMetastoreDatabaseFactory.java:52)
at
gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(TestMetastoreDatabaseFactory.java:47)
at
gobblin.metastore.DatabaseJobHistoryStoreTest.setUp(DatabaseJobHistoryStoreTest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
I was able to resolve this by downloading the mysql from here
https://downloads.mysql.com/archives/get/file/mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz
And copying it at the location /home/vicky/.embedmysql/MySQL-5.6/.
I did not investigate if we could fix it by some configuration, I just
fixed it in this way.
2) Next failure is coming as
********************************************************************************************************
:gobblin-yarn:test
:gobblin-modules:gobblin-kafka-08:compileTestJava FAILED
> Building 75% > :gobblin-runtime:test > 58 tests completed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':gobblin-modules:gobblin-kafka-08:compileTestJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or
--debug option to get more log output.
BUILD FAILED
Total time: 5 mins 19.311 secs
********************************************************************************************************
Since I have blacklisted the kafka-08 in the setting.gradle so it might be
giving the compilation error. Can someone try to run the build with the
kafak-09 flavor on gobblin0.9 and let me know?
I am expecting the build/run to work smoothly however it is not working as
expected.
Thanks,
Vicky
Re: Gobbin build failure when using custom flavor
Posted by Vicky Kak <vi...@gmail.com>.
Yep I think it is an issue with the older tag as mentioned. I don't
recollect if this happened on the truck too.
On Fri, Sep 8, 2017 at 5:34 AM, Abhishek Tiwari <ab...@apache.org> wrote:
> Is this still an issue?
>
> On Wed, Aug 9, 2017 at 5:51 AM, Vicky Kak <vi...@gmail.com> wrote:
>
>> Hi All,
>>
>> I have been building the tag gobblin_0.9.0 with kafka 9, here are the
>> changes that I did
>>
>> 1)
>> vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin$ git diff
>> diff --git a/gobblin-distribution/gobblin-flavor-custom.gradle
>> b/gobblin-distribution/gobblin-flavor-custom.gradle
>> index 5b363bd45..a890c4944 100644
>> --- a/gobblin-distribution/gobblin-flavor-custom.gradle
>> +++ b/gobblin-distribution/gobblin-flavor-custom.gradle
>> @@ -1,6 +1,6 @@
>> dependencies {
>> // Example jobs
>> // compile project(':gobblin-example')
>> -
>> + compile project(':gobblin-modules:gobblin-kafka-09')
>> }
>>
>> diff --git a/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep
>> b/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep
>> deleted file mode 100644
>> index e69de29bb..000000000
>> diff --git a/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep
>> b/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep
>> deleted file mode 100644
>> index e69de29bb..000000000
>> diff --git a/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep
>> b/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep
>> deleted file mode 100644
>> index e69de29bb..000000000
>> diff --git a/settings.gradle b/settings.gradle
>> index c63463e0f..ac8db6940 100644
>> --- a/settings.gradle
>> +++ b/settings.gradle
>> @@ -41,7 +41,7 @@ def modules = ['gobblin-admin',
>> // Disable jacoco for now as Kafka 0.8 is the default version and jacoco
>> does not like the same classes
>> // being declared in different modules
>> def jacocoBlacklist = new HashSet([
>> - "gobblin-modules:gobblin-kafka-09"
>> + "gobblin-modules:gobblin-kafka-08"
>> ])
>>
>> modules.each { module ->
>>
>> 2) I also added the gobblin-flavor-custom.gradle in the root directory
>> which is vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin
>>
>> Finally I called
>> ./gradlew -PgobblinFlavor=custom assemble
>>
>> It created the gobblin distribution with the Kakfka0.9.
>>
>> When I run the build which includes the test I got these issues
>>
>> 1) gobblin.metastore.DatabaseJobHistoryStoreV100Test:setUp failed with
>> following error
>>
>> de.flapdoodle.embed.process.exceptions.DistributionException:
>> java.io.IOException: File /home/vicky/.embedmysql/MySQL-
>> 5.6/mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz
>> at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.
>> java:69)
>> at de.flapdoodle.embed.process.runtime.Starter.prepare(Starter.
>> java:49)
>> at com.wix.mysql.EmbeddedMysql.<init>(EmbeddedMysql.java:39)
>> at com.wix.mysql.EmbeddedMysql$Builder.start(EmbeddedMysql.java:131)
>> at gobblin.metastore.testing.TestMetastoreDatabaseServer.<init>
>> (TestMetastoreDatabaseServer.java:94)
>> at gobblin.metastore.testing.TestMetastoreDatabaseFactory.ensur
>> eDatabaseExists(TestMetastoreDatabaseFactory.java:72)
>> at gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(T
>> estMetastoreDatabaseFactory.java:52)
>> at gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(T
>> estMetastoreDatabaseFactory.java:47)
>> at gobblin.metastore.DatabaseJobHistoryStoreTest.setUp(Database
>> JobHistoryStoreTest.java:68)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> I was able to resolve this by downloading the mysql from here
>> https://downloads.mysql.com/archives/get/file/mysql-5.6.24-
>> linux-glibc2.5-x86_64.tar.gz
>>
>> And copying it at the location /home/vicky/.embedmysql/MySQL-5.6/.
>>
>> I did not investigate if we could fix it by some configuration, I just
>> fixed it in this way.
>>
>> 2) Next failure is coming as
>>
>> ************************************************************
>> ********************************************
>> :gobblin-yarn:test
>> :gobblin-modules:gobblin-kafka-08:compileTestJava FAILED
>> > Building 75% > :gobblin-runtime:test > 58 tests completed
>>
>> FAILURE: Build failed with an exception.
>>
>> * What went wrong:
>> Execution failed for task ':gobblin-modules:gobblin-kafk
>> a-08:compileTestJava'.
>> > Compilation failed; see the compiler error output for details.
>>
>> * Try:
>> Run with --stacktrace option to get the stack trace. Run with --info or
>> --debug option to get more log output.
>>
>> BUILD FAILED
>>
>> Total time: 5 mins 19.311 secs
>> ************************************************************
>> ********************************************
>>
>> Since I have blacklisted the kafka-08 in the setting.gradle so it might
>> be giving the compilation error. Can someone try to run the build with the
>> kafak-09 flavor on gobblin0.9 and let me know?
>> I am expecting the build/run to work smoothly however it is not working
>> as expected.
>>
>> Thanks,
>> Vicky
>>
>
>
Re: Gobbin build failure when using custom flavor
Posted by Abhishek Tiwari <ab...@apache.org>.
Is this still an issue?
On Wed, Aug 9, 2017 at 5:51 AM, Vicky Kak <vi...@gmail.com> wrote:
> Hi All,
>
> I have been building the tag gobblin_0.9.0 with kafka 9, here are the
> changes that I did
>
> 1)
> vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin$ git diff
> diff --git a/gobblin-distribution/gobblin-flavor-custom.gradle
> b/gobblin-distribution/gobblin-flavor-custom.gradle
> index 5b363bd45..a890c4944 100644
> --- a/gobblin-distribution/gobblin-flavor-custom.gradle
> +++ b/gobblin-distribution/gobblin-flavor-custom.gradle
> @@ -1,6 +1,6 @@
> dependencies {
> // Example jobs
> // compile project(':gobblin-example')
> -
> + compile project(':gobblin-modules:gobblin-kafka-09')
> }
>
> diff --git a/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep
> b/gobblin-runtime/src/test/resources/runtime_test/state_store/.keep
> deleted file mode 100644
> index e69de29bb..000000000
> diff --git a/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep
> b/gobblin-runtime/src/test/resources/runtime_test/writer_output/.keep
> deleted file mode 100644
> index e69de29bb..000000000
> diff --git a/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep
> b/gobblin-runtime/src/test/resources/runtime_test/writer_staging/.keep
> deleted file mode 100644
> index e69de29bb..000000000
> diff --git a/settings.gradle b/settings.gradle
> index c63463e0f..ac8db6940 100644
> --- a/settings.gradle
> +++ b/settings.gradle
> @@ -41,7 +41,7 @@ def modules = ['gobblin-admin',
> // Disable jacoco for now as Kafka 0.8 is the default version and jacoco
> does not like the same classes
> // being declared in different modules
> def jacocoBlacklist = new HashSet([
> - "gobblin-modules:gobblin-kafka-09"
> + "gobblin-modules:gobblin-kafka-08"
> ])
>
> modules.each { module ->
>
> 2) I also added the gobblin-flavor-custom.gradle in the root directory
> which is vicky@vicky-Latitude-E5570:~/git/apache/incubator-gobblin
>
> Finally I called
> ./gradlew -PgobblinFlavor=custom assemble
>
> It created the gobblin distribution with the Kakfka0.9.
>
> When I run the build which includes the test I got these issues
>
> 1) gobblin.metastore.DatabaseJobHistoryStoreV100Test:setUp failed with
> following error
>
> de.flapdoodle.embed.process.exceptions.DistributionException:
> java.io.IOException: File /home/vicky/.embedmysql/MySQL-
> 5.6/mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz
> at de.flapdoodle.embed.process.runtime.Starter.prepare(
> Starter.java:69)
> at de.flapdoodle.embed.process.runtime.Starter.prepare(
> Starter.java:49)
> at com.wix.mysql.EmbeddedMysql.<init>(EmbeddedMysql.java:39)
> at com.wix.mysql.EmbeddedMysql$Builder.start(EmbeddedMysql.java:131)
> at gobblin.metastore.testing.TestMetastoreDatabaseServer.<init>(
> TestMetastoreDatabaseServer.java:94)
> at gobblin.metastore.testing.TestMetastoreDatabaseFactory.
> ensureDatabaseExists(TestMetastoreDatabaseFactory.java:72)
> at gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(
> TestMetastoreDatabaseFactory.java:52)
> at gobblin.metastore.testing.TestMetastoreDatabaseFactory.get(
> TestMetastoreDatabaseFactory.java:47)
> at gobblin.metastore.DatabaseJobHistoryStoreTest.setUp(
> DatabaseJobHistoryStoreTest.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> I was able to resolve this by downloading the mysql from here
> https://downloads.mysql.com/archives/get/file/mysql-5.6.
> 24-linux-glibc2.5-x86_64.tar.gz
>
> And copying it at the location /home/vicky/.embedmysql/MySQL-5.6/.
>
> I did not investigate if we could fix it by some configuration, I just
> fixed it in this way.
>
> 2) Next failure is coming as
>
> ************************************************************
> ********************************************
> :gobblin-yarn:test
> :gobblin-modules:gobblin-kafka-08:compileTestJava FAILED
> > Building 75% > :gobblin-runtime:test > 58 tests completed
>
> FAILURE: Build failed with an exception.
>
> * What went wrong:
> Execution failed for task ':gobblin-modules:gobblin-
> kafka-08:compileTestJava'.
> > Compilation failed; see the compiler error output for details.
>
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or
> --debug option to get more log output.
>
> BUILD FAILED
>
> Total time: 5 mins 19.311 secs
> ************************************************************
> ********************************************
>
> Since I have blacklisted the kafka-08 in the setting.gradle so it might be
> giving the compilation error. Can someone try to run the build with the
> kafak-09 flavor on gobblin0.9 and let me know?
> I am expecting the build/run to work smoothly however it is not working as
> expected.
>
> Thanks,
> Vicky
>