You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Thomas Weise <th...@apache.org> on 2018/08/15 05:54:42 UTC

Publishing release artifacts to custom artifactory

Hi,

I wanted to see if anyone has used the Gradle build to publish the
release/snapshot artifacts to custom repo?

So far I found the following (attempting to publish a particular Maven
artifact):

./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing
-PdistMgmtSnapshotsUrl=https://somecustomrepo/libs-snapshots --no-parallel
--no-daemon

It gets to the point of attempting deploy to the remote repo, but fails due
to hardcoded server IDs:

https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817

When the version isn't -SNAPSHOT, then I guess it would instead be
-PisRelease, the build will fail because it expects gpg to run, which I
don't need for what I'm trying to do.

Have others attempted third party builds for the Python SDK or Docker
images? Any gotchas to share?

Thanks,
Thomas

Re: Publishing release artifacts to custom artifactory

Posted by Alexey Romanenko <ar...@gmail.com>.
Thomas, thanks, looks great. Do you think we have to add this command to “Contribution Guide”?

Lukasz, yes, "-Poffline-repository"  can be omitted in this case. I don’t remember why I added this =)

> On 24 Aug 2018, at 22:15, Thomas Weise <th...@apache.org> wrote:
> 
> Alexey, publishing to custom repo with authentication is now possible, see https://github.com/apache/beam/pull/6230 <https://github.com/apache/beam/pull/6230> with example.
> 
> 
> On Fri, Aug 24, 2018 at 1:08 PM Lukasz Cwik <lcwik@google.com <ma...@google.com>> wrote:
> "-Poffline-repository" controls the addition of another maven repo to read dependencies from, it doesn't impact project publishing and shouldn't be needed.
> 
> On Fri, Aug 24, 2018 at 5:28 AM Alexey Romanenko <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
> Maybe my answer is not 100% relevant to initial topic (sorry for that in advance) but it took me quite a time to find out how to properly install artefacts into local maven repository with gradle.
> Finally, I came to this command (additional flags are skipped for the sake of simplicity).
> 
> ./gradlew -Ppublishing -Poffline-repository=file:///<path_to_local_repository> <> --no-parallel -PdistMgmtSnapshotsUrl=file:/// <><path_to_local_repository> -p <path_to_beam_module> publish
> 
> For example,I used this command to publish KinesisIO artifacts on local maven to test my changes against external Beam-based application:
> 
> ./gradlew -Ppublishing -Poffline-repository=file:///Users/aromanenko/.m2/repository/ <> --no-parallel -PdistMgmtSnapshotsUrl=file:///Users/aromanenko/.m2/repository/ <> -p sdks/java/io/kinesis/ publish
> 
> Not sure, if this way is fully correct, but, at least, it's worked for me =)
> 
>> On 15 Aug 2018, at 16:32, Thomas Weise <thw@apache.org <ma...@apache.org>> wrote:
>> 
>> Hey JB,
>> 
>> I think it has to be -PisRelease=false (otherwise the publish task isn't activated).
>> 
>> For -SNAPSHOT, I'm getting further with https://github.com/apache/beam/pull/6230 <https://github.com/apache/beam/pull/6230>
>> 
>> I'm now looking for a good way to allow skipping gpg (when the version isn't -SNAPSHOT). Currently, it will run whenever the isRelease property is present.
>> 
>> Thanks
>> 
>> 
>> On Wed, Aug 15, 2018 at 4:42 AM Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>> Hi Thomas,
>> 
>> yes, I did on a local Nexus repository.
>> 
>> Did you try with -DisRelease=false ?
>> 
>> Regards
>> JB
>> 
>> On 15/08/2018 07:54, Thomas Weise wrote:
>> > Hi,
>> > 
>> > I wanted to see if anyone has used the Gradle build to publish the
>> > release/snapshot artifacts to custom repo?
>> > 
>> > So far I found the following (attempting to publish a particular Maven
>> > artifact):
>> > 
>> > ./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing
>> > -PdistMgmtSnapshotsUrl=https://somecustomrepo/libs-snapshots <https://somecustomrepo/libs-snapshots>
>> > --no-parallel --no-daemon
>> > 
>> > It gets to the point of attempting deploy to the remote repo, but fails
>> > due to hardcoded server IDs:
>> > 
>> > https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817 <https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817>
>> > 
>> > When the version isn't -SNAPSHOT, then I guess it would instead be
>> > -PisRelease, the build will fail because it expects gpg to run, which I
>> > don't need for what I'm trying to do.
>> > 
>> > Have others attempted third party builds for the Python SDK or Docker
>> > images? Any gotchas to share?
>> > 
>> > Thanks,
>> > Thomas
>> > 
>> > 
>> 
>> -- 
>> Jean-Baptiste Onofré
>> jbonofre@apache.org <ma...@apache.org>
>> http://blog.nanthrax.net <http://blog.nanthrax.net/>
>> Talend - http://www.talend.com <http://www.talend.com/>
>

Re: Publishing release artifacts to custom artifactory

Posted by Thomas Weise <th...@apache.org>.
Alexey, publishing to custom repo with authentication is now possible, see
https://github.com/apache/beam/pull/6230 with example.


On Fri, Aug 24, 2018 at 1:08 PM Lukasz Cwik <lc...@google.com> wrote:

> *"-Poffline-repository" *controls the addition of another maven repo to
> read dependencies from, it doesn't impact project publishing and shouldn't
> be needed.
>
> On Fri, Aug 24, 2018 at 5:28 AM Alexey Romanenko <ar...@gmail.com>
> wrote:
>
>> Maybe my answer is not 100% relevant to initial topic (sorry for that in
>> advance) but it took me quite a time to find out how to properly install
>> artefacts into local maven repository with gradle.
>> Finally, I came to this command (additional flags are skipped for the
>> sake of simplicity).
>>
>> *./gradlew -Ppublishing
>> -Poffline-repository=file:///<path_to_local_repository> --no-parallel
>> -PdistMgmtSnapshotsUrl=file:///**<path_to_local_repository>** -p
>> <path_to_beam_module> publish*
>>
>> For example,I used this command to publish KinesisIO artifacts on local
>> maven to test my changes against external Beam-based application:
>>
>> *./gradlew -Ppublishing
>> -Poffline-repository=file:///Users/aromanenko/.m2/repository/ --no-parallel
>> -PdistMgmtSnapshotsUrl=file:///Users/aromanenko/.m2/repository/ -p
>> sdks/java/io/kinesis/ publish*
>>
>> Not sure, if this way is fully correct, but, at least, it's worked for me
>> =)
>>
>> On 15 Aug 2018, at 16:32, Thomas Weise <th...@apache.org> wrote:
>>
>> Hey JB,
>>
>> I think it has to be -PisRelease=false (otherwise the publish task isn't
>> activated).
>>
>> For -SNAPSHOT, I'm getting further with
>> https://github.com/apache/beam/pull/6230
>>
>> I'm now looking for a good way to allow skipping gpg (when the version
>> isn't -SNAPSHOT). Currently, it will run whenever the isRelease property is
>> present.
>>
>> Thanks
>>
>>
>> On Wed, Aug 15, 2018 at 4:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Hi Thomas,
>>>
>>> yes, I did on a local Nexus repository.
>>>
>>> Did you try with -DisRelease=false ?
>>>
>>> Regards
>>> JB
>>>
>>> On 15/08/2018 07:54, Thomas Weise wrote:
>>> > Hi,
>>> >
>>> > I wanted to see if anyone has used the Gradle build to publish the
>>> > release/snapshot artifacts to custom repo?
>>> >
>>> > So far I found the following (attempting to publish a particular Maven
>>> > artifact):
>>> >
>>> > ./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing
>>> > -PdistMgmtSnapshotsUrl=https://somecustomrepo/libs-snapshots
>>> > --no-parallel --no-daemon
>>> >
>>> > It gets to the point of attempting deploy to the remote repo, but fails
>>> > due to hardcoded server IDs:
>>> >
>>> >
>>> https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817
>>> >
>>> > When the version isn't -SNAPSHOT, then I guess it would instead be
>>> > -PisRelease, the build will fail because it expects gpg to run, which I
>>> > don't need for what I'm trying to do.
>>> >
>>> > Have others attempted third party builds for the Python SDK or Docker
>>> > images? Any gotchas to share?
>>> >
>>> > Thanks,
>>> > Thomas
>>> >
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>>

Re: Publishing release artifacts to custom artifactory

Posted by Lukasz Cwik <lc...@google.com>.
*"-Poffline-repository" *controls the addition of another maven repo to
read dependencies from, it doesn't impact project publishing and shouldn't
be needed.

On Fri, Aug 24, 2018 at 5:28 AM Alexey Romanenko <ar...@gmail.com>
wrote:

> Maybe my answer is not 100% relevant to initial topic (sorry for that in
> advance) but it took me quite a time to find out how to properly install
> artefacts into local maven repository with gradle.
> Finally, I came to this command (additional flags are skipped for the sake
> of simplicity).
>
> *./gradlew -Ppublishing
> -Poffline-repository=file:///<path_to_local_repository> --no-parallel
> -PdistMgmtSnapshotsUrl=file:///**<path_to_local_repository>** -p
> <path_to_beam_module> publish*
>
> For example,I used this command to publish KinesisIO artifacts on local
> maven to test my changes against external Beam-based application:
>
> *./gradlew -Ppublishing
> -Poffline-repository=file:///Users/aromanenko/.m2/repository/ --no-parallel
> -PdistMgmtSnapshotsUrl=file:///Users/aromanenko/.m2/repository/ -p
> sdks/java/io/kinesis/ publish*
>
> Not sure, if this way is fully correct, but, at least, it's worked for me
> =)
>
> On 15 Aug 2018, at 16:32, Thomas Weise <th...@apache.org> wrote:
>
> Hey JB,
>
> I think it has to be -PisRelease=false (otherwise the publish task isn't
> activated).
>
> For -SNAPSHOT, I'm getting further with
> https://github.com/apache/beam/pull/6230
>
> I'm now looking for a good way to allow skipping gpg (when the version
> isn't -SNAPSHOT). Currently, it will run whenever the isRelease property is
> present.
>
> Thanks
>
>
> On Wed, Aug 15, 2018 at 4:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Thomas,
>>
>> yes, I did on a local Nexus repository.
>>
>> Did you try with -DisRelease=false ?
>>
>> Regards
>> JB
>>
>> On 15/08/2018 07:54, Thomas Weise wrote:
>> > Hi,
>> >
>> > I wanted to see if anyone has used the Gradle build to publish the
>> > release/snapshot artifacts to custom repo?
>> >
>> > So far I found the following (attempting to publish a particular Maven
>> > artifact):
>> >
>> > ./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing
>> > -PdistMgmtSnapshotsUrl=https://somecustomrepo/libs-snapshots
>> > --no-parallel --no-daemon
>> >
>> > It gets to the point of attempting deploy to the remote repo, but fails
>> > due to hardcoded server IDs:
>> >
>> >
>> https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817
>> >
>> > When the version isn't -SNAPSHOT, then I guess it would instead be
>> > -PisRelease, the build will fail because it expects gpg to run, which I
>> > don't need for what I'm trying to do.
>> >
>> > Have others attempted third party builds for the Python SDK or Docker
>> > images? Any gotchas to share?
>> >
>> > Thanks,
>> > Thomas
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>

Re: Publishing release artifacts to custom artifactory

Posted by Alexey Romanenko <ar...@gmail.com>.
Maybe my answer is not 100% relevant to initial topic (sorry for that in advance) but it took me quite a time to find out how to properly install artefacts into local maven repository with gradle.
Finally, I came to this command (additional flags are skipped for the sake of simplicity).

./gradlew -Ppublishing -Poffline-repository=file:///<path_to_local_repository> --no-parallel -PdistMgmtSnapshotsUrl=file:///<path_to_local_repository> -p <path_to_beam_module> publish

For example,I used this command to publish KinesisIO artifacts on local maven to test my changes against external Beam-based application:

./gradlew -Ppublishing -Poffline-repository=file:///Users/aromanenko/.m2/repository/ --no-parallel -PdistMgmtSnapshotsUrl=file:///Users/aromanenko/.m2/repository/ -p sdks/java/io/kinesis/ publish

Not sure, if this way is fully correct, but, at least, it's worked for me =)

> On 15 Aug 2018, at 16:32, Thomas Weise <th...@apache.org> wrote:
> 
> Hey JB,
> 
> I think it has to be -PisRelease=false (otherwise the publish task isn't activated).
> 
> For -SNAPSHOT, I'm getting further with https://github.com/apache/beam/pull/6230 <https://github.com/apache/beam/pull/6230>
> 
> I'm now looking for a good way to allow skipping gpg (when the version isn't -SNAPSHOT). Currently, it will run whenever the isRelease property is present.
> 
> Thanks
> 
> 
> On Wed, Aug 15, 2018 at 4:42 AM Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> Hi Thomas,
> 
> yes, I did on a local Nexus repository.
> 
> Did you try with -DisRelease=false ?
> 
> Regards
> JB
> 
> On 15/08/2018 07:54, Thomas Weise wrote:
> > Hi,
> > 
> > I wanted to see if anyone has used the Gradle build to publish the
> > release/snapshot artifacts to custom repo?
> > 
> > So far I found the following (attempting to publish a particular Maven
> > artifact):
> > 
> > ./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing
> > -PdistMgmtSnapshotsUrl=https://somecustomrepo/libs-snapshots <https://somecustomrepo/libs-snapshots>
> > --no-parallel --no-daemon
> > 
> > It gets to the point of attempting deploy to the remote repo, but fails
> > due to hardcoded server IDs:
> > 
> > https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817 <https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817>
> > 
> > When the version isn't -SNAPSHOT, then I guess it would instead be
> > -PisRelease, the build will fail because it expects gpg to run, which I
> > don't need for what I'm trying to do.
> > 
> > Have others attempted third party builds for the Python SDK or Docker
> > images? Any gotchas to share?
> > 
> > Thanks,
> > Thomas
> > 
> > 
> 
> -- 
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> Talend - http://www.talend.com <http://www.talend.com/>

Re: Publishing release artifacts to custom artifactory

Posted by Thomas Weise <th...@apache.org>.
Hey JB,

I think it has to be -PisRelease=false (otherwise the publish task isn't
activated).

For -SNAPSHOT, I'm getting further with
https://github.com/apache/beam/pull/6230

I'm now looking for a good way to allow skipping gpg (when the version
isn't -SNAPSHOT). Currently, it will run whenever the isRelease property is
present.

Thanks


On Wed, Aug 15, 2018 at 4:42 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi Thomas,
>
> yes, I did on a local Nexus repository.
>
> Did you try with -DisRelease=false ?
>
> Regards
> JB
>
> On 15/08/2018 07:54, Thomas Weise wrote:
> > Hi,
> >
> > I wanted to see if anyone has used the Gradle build to publish the
> > release/snapshot artifacts to custom repo?
> >
> > So far I found the following (attempting to publish a particular Maven
> > artifact):
> >
> > ./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing
> > -PdistMgmtSnapshotsUrl=https://somecustomrepo/libs-snapshots
> > --no-parallel --no-daemon
> >
> > It gets to the point of attempting deploy to the remote repo, but fails
> > due to hardcoded server IDs:
> >
> >
> https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817
> >
> > When the version isn't -SNAPSHOT, then I guess it would instead be
> > -PisRelease, the build will fail because it expects gpg to run, which I
> > don't need for what I'm trying to do.
> >
> > Have others attempted third party builds for the Python SDK or Docker
> > images? Any gotchas to share?
> >
> > Thanks,
> > Thomas
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Publishing release artifacts to custom artifactory

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Thomas,

yes, I did on a local Nexus repository.

Did you try with -DisRelease=false ?

Regards
JB

On 15/08/2018 07:54, Thomas Weise wrote:
> Hi,
> 
> I wanted to see if anyone has used the Gradle build to publish the
> release/snapshot artifacts to custom repo?
> 
> So far I found the following (attempting to publish a particular Maven
> artifact):
> 
> ./gradlew :beam-runners-flink_2.11-job-server:publish -Ppublishing
> -PdistMgmtSnapshotsUrl=https://somecustomrepo/libs-snapshots
> --no-parallel --no-daemon
> 
> It gets to the point of attempting deploy to the remote repo, but fails
> due to hardcoded server IDs:
> 
> https://github.com/apache/beam/blob/b82263390af57a2d52c5d7b257470bc837606e4e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L817
> 
> When the version isn't -SNAPSHOT, then I guess it would instead be
> -PisRelease, the build will fail because it expects gpg to run, which I
> don't need for what I'm trying to do.
> 
> Have others attempted third party builds for the Python SDK or Docker
> images? Any gotchas to share?
> 
> Thanks,
> Thomas
> 
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com