You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Mike Pedersen <mi...@mikepedersen.dk> on 2019/02/07 14:06:14 UTC
Installing Apache Beam locally
Hi all. I have made some changes to Beam's Java SDK locally and would like
to install that version into my local repository. Some of the modules'
install tasks fail, so I disabled those, eventually leading me to a install
command like this:
./gradlew install \
-x :beam-sdks-java-maven-archetypes-examples:install \
-x :beam-runners-reference-job-server:install \
-x :beam-runners-flink-1.6-job-server:install \
-x :beam-runners-flink-1.7-job-server:install \
-x :beam-runners-flink_2.11-job-server:install
This command completes without error, but seems to only install unshaded
dependencies unlike the shaded jars in the maven repository:
$ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.9.0-SNAPSHOT/
beam-sdks-java-core-2.9.0-SNAPSHOT-tests-unshaded.jar
beam-sdks-java-core-2.9.0-SNAPSHOT-unshaded.jar
beam-sdks-java-core-2.9.0-SNAPSHOT.pom
maven-metadata-local.xml
Whereas the ones from the maven repository is like this instead (2.8.0, but
I assume it should be the same for 2.9.0):
$ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.8.0/
_remote.repositories
beam-sdks-java-core-2.8.0.jar
beam-sdks-java-core-2.8.0.jar.sha1
beam-sdks-java-core-2.8.0.pom
beam-sdks-java-core-2.8.0.pom.sha1
How do I get gradle to install the shaded jars like the ones found in the
Beam's maven repository?
Thanks in advance,
Mike
Re: Installing Apache Beam locally
Posted by Mike Pedersen <mi...@mikepedersen.dk>.
Thanks a lot. That did the trick.
One of the reasons I wanted this was to build it with jitpack.io. For
anyone else that might also want this, here is my jitpack.yml file:
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
install:
- >
./gradlew -Ppublishing -PnoSigning publishMavenJavaPublicationToMavenLocal
-x
beam-sdks-java-maven-archetypes-examples:publishMavenJavaPublicationToMavenLocal
Yes, the license has to be there or the license checker will complain. The
beam-sdks-java-maven-archetypes-examples module is disabled as it uses some
commands not found in the jitpack.io vm.
Mike
Den tor. 7. feb. 2019 kl. 15.24 skrev Kenneth Knowles <ke...@apache.org>:
> It is a bit of a funky invocation, and our config makes it a little
> stranger yet. I think you want
>
> ./gradlew -Ppublishing
> -PnoSigning publishMavenJavaPublicationToMavenLocal
>
> -Ppublishing is in our config to enable/disable the publishing plugin
> -PnoSigning turns off signing which is on by default but you probably
> don't want
> The rest is how the publishing plugin does its thing, a concatenation of:
> publish + <what to publish> + Publication To + <where to publish>
>
> I didn't actually realize the `gradle install` did anything. But I think
> it is worth making it so `gradle install` does exactly the above.
>
> Kenn
>
> On Thu, Feb 7, 2019 at 6:06 AM Mike Pedersen <mi...@mikepedersen.dk> wrote:
>
>> Hi all. I have made some changes to Beam's Java SDK locally and would
>> like to install that version into my local repository. Some of the modules'
>> install tasks fail, so I disabled those, eventually leading me to a install
>> command like this:
>>
>> ./gradlew install \
>> -x :beam-sdks-java-maven-archetypes-examples:install \
>> -x :beam-runners-reference-job-server:install \
>> -x :beam-runners-flink-1.6-job-server:install \
>> -x :beam-runners-flink-1.7-job-server:install \
>> -x :beam-runners-flink_2.11-job-server:install
>>
>> This command completes without error, but seems to only install unshaded
>> dependencies unlike the shaded jars in the maven repository:
>>
>> $ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.9.0-SNAPSHOT/
>> beam-sdks-java-core-2.9.0-SNAPSHOT-tests-unshaded.jar
>> beam-sdks-java-core-2.9.0-SNAPSHOT-unshaded.jar
>> beam-sdks-java-core-2.9.0-SNAPSHOT.pom
>> maven-metadata-local.xml
>>
>> Whereas the ones from the maven repository is like this instead (2.8.0,
>> but I assume it should be the same for 2.9.0):
>>
>> $ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.8.0/
>> _remote.repositories
>> beam-sdks-java-core-2.8.0.jar
>> beam-sdks-java-core-2.8.0.jar.sha1
>> beam-sdks-java-core-2.8.0.pom
>> beam-sdks-java-core-2.8.0.pom.sha1
>>
>> How do I get gradle to install the shaded jars like the ones found in the
>> Beam's maven repository?
>>
>> Thanks in advance,
>> Mike
>>
>
Re: Installing Apache Beam locally
Posted by Kenneth Knowles <ke...@apache.org>.
It is a bit of a funky invocation, and our config makes it a little
stranger yet. I think you want
./gradlew -Ppublishing
-PnoSigning publishMavenJavaPublicationToMavenLocal
-Ppublishing is in our config to enable/disable the publishing plugin
-PnoSigning turns off signing which is on by default but you probably don't
want
The rest is how the publishing plugin does its thing, a concatenation of:
publish + <what to publish> + Publication To + <where to publish>
I didn't actually realize the `gradle install` did anything. But I think it
is worth making it so `gradle install` does exactly the above.
Kenn
On Thu, Feb 7, 2019 at 6:06 AM Mike Pedersen <mi...@mikepedersen.dk> wrote:
> Hi all. I have made some changes to Beam's Java SDK locally and would like
> to install that version into my local repository. Some of the modules'
> install tasks fail, so I disabled those, eventually leading me to a install
> command like this:
>
> ./gradlew install \
> -x :beam-sdks-java-maven-archetypes-examples:install \
> -x :beam-runners-reference-job-server:install \
> -x :beam-runners-flink-1.6-job-server:install \
> -x :beam-runners-flink-1.7-job-server:install \
> -x :beam-runners-flink_2.11-job-server:install
>
> This command completes without error, but seems to only install unshaded
> dependencies unlike the shaded jars in the maven repository:
>
> $ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.9.0-SNAPSHOT/
> beam-sdks-java-core-2.9.0-SNAPSHOT-tests-unshaded.jar
> beam-sdks-java-core-2.9.0-SNAPSHOT-unshaded.jar
> beam-sdks-java-core-2.9.0-SNAPSHOT.pom
> maven-metadata-local.xml
>
> Whereas the ones from the maven repository is like this instead (2.8.0,
> but I assume it should be the same for 2.9.0):
>
> $ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.8.0/
> _remote.repositories
> beam-sdks-java-core-2.8.0.jar
> beam-sdks-java-core-2.8.0.jar.sha1
> beam-sdks-java-core-2.8.0.pom
> beam-sdks-java-core-2.8.0.pom.sha1
>
> How do I get gradle to install the shaded jars like the ones found in the
> Beam's maven repository?
>
> Thanks in advance,
> Mike
>
Re: Installing Apache Beam locally
Posted by Kenneth Knowles <ke...@apache.org>.
It is a bit of a funky invocation, and our config makes it a little
stranger yet. I think you want
./gradlew -Ppublishing
-PnoSigning publishMavenJavaPublicationToMavenLocal
-Ppublishing is in our config to enable/disable the publishing plugin
-PnoSigning turns off signing which is on by default but you probably don't
want
The rest is how the publishing plugin does its thing, a concatenation of:
publish + <what to publish> + Publication To + <where to publish>
I didn't actually realize the `gradle install` did anything. But I think it
is worth making it so `gradle install` does exactly the above.
Kenn
On Thu, Feb 7, 2019 at 6:06 AM Mike Pedersen <mi...@mikepedersen.dk> wrote:
> Hi all. I have made some changes to Beam's Java SDK locally and would like
> to install that version into my local repository. Some of the modules'
> install tasks fail, so I disabled those, eventually leading me to a install
> command like this:
>
> ./gradlew install \
> -x :beam-sdks-java-maven-archetypes-examples:install \
> -x :beam-runners-reference-job-server:install \
> -x :beam-runners-flink-1.6-job-server:install \
> -x :beam-runners-flink-1.7-job-server:install \
> -x :beam-runners-flink_2.11-job-server:install
>
> This command completes without error, but seems to only install unshaded
> dependencies unlike the shaded jars in the maven repository:
>
> $ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.9.0-SNAPSHOT/
> beam-sdks-java-core-2.9.0-SNAPSHOT-tests-unshaded.jar
> beam-sdks-java-core-2.9.0-SNAPSHOT-unshaded.jar
> beam-sdks-java-core-2.9.0-SNAPSHOT.pom
> maven-metadata-local.xml
>
> Whereas the ones from the maven repository is like this instead (2.8.0,
> but I assume it should be the same for 2.9.0):
>
> $ ls ~/.m2/repository/org/apache/beam/beam-sdks-java-core/2.8.0/
> _remote.repositories
> beam-sdks-java-core-2.8.0.jar
> beam-sdks-java-core-2.8.0.jar.sha1
> beam-sdks-java-core-2.8.0.pom
> beam-sdks-java-core-2.8.0.pom.sha1
>
> How do I get gradle to install the shaded jars like the ones found in the
> Beam's maven repository?
>
> Thanks in advance,
> Mike
>