You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Sander Verhagen <sa...@sanderverhagen.net> on 2018/12/11 21:19:32 UTC
Maven builds stalling on Bitbucket
Hi all,
Need some help! My current gig is running its Maven builds on Bitbucket. On a regular basis, builds stall while downloading dependencies (I'd say 5%~10% of all builds). It looks a lot like [1]. The proposed solution (maven.wagon.http.pool=false) does not show any noticeable improvement. Atlassian declined to investigate, even though this only ever happens on Bitbucket (local builds have never been seen having this problem).
I have included some logging (-X) from just before the stall. The stalls can happen anywhere dependencies are downloaded, so seemingly unrelated to this particular goal.
I would love to get some suggestions on how to fix this, or at least on how to troubleshoot it further.
Thank you!
[1] https://bitbucket.org/site/master/issues/13988/pipelines-kills-idle-maven-connections
Sander Verhagen
[ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
[DEBUG] -----------------------------------------------------------------------
[DEBUG] Goal: com.github.warmuuh:pactbroker-maven-plugin:0.0.13:upload-pacts (upload-pacts)
[DEBUG] Style: Regular
[DEBUG] Configuration: <?xml version="1.0" encoding="UTF-8"?>
<configuration>
<... snip ...>
</configuration>
[DEBUG] =======================================================================
[DEBUG] Could not find metadata commons-io:commons-io/maven-metadata.xml in local (/root/.m2/repository)
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://jitpack.io
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://repo.spring.io/libs-snapshot
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://jitpack.io
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
[INFO] Downloading from some_repo_snapshot: http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/commons-io/commons-io/maven-metadata.xml
[INFO] Downloading from jitpack.io: https://jitpack.io/commons-io/commons-io/maven-metadata.xml
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://repo.spring.io/libs-snapshot
[INFO] Downloading from spring-snapshots: http://repo.spring.io/libs-snapshot/commons-io/commons-io/maven-metadata.xml
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
[INFO] Downloading from some_repo_release: http://some_repo_release/nexus/content/repositories/some_repo_release/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://oss.sonatype.org/content/repositories/snapshots
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://oss.sonatype.org/content/repositories/snapshots
[INFO] Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://clojars.org/repo
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://clojars.org/repo
[INFO] Downloading from clojars: http://clojars.org/repo/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
// nothing new happened here for 20 minutes until Bitbucket (rightfully) killed the build
Re: Maven builds stalling on Bitbucket
Posted by Sander Verhagen <sa...@sanderverhagen.net>.
Michael "et al",
At this point I want to thank you very much for great work on this issue. I removed the other hacks from my build configuration, and now have it running with the newer version. I have not seen the error for a while. It took me a while and some unease to bring me to that conclusion, though, because I strongly believe I saw the issue on my test branch again, short after applying these changes. After that, I have been running with -X for days, in the hopes of giving you an equally thorough updated report as the stuff you did, but to no avail. There is only so many build minutes (dollars) I can spend on this (rerunning the same build over and over with -X) and I may very well be chasing some red herring. But in any case, it brought our presence on Bitbucket back from "unattainable" to "somewhat acceptable". With Bitbucket, that's all we can wish for. Thanks again, though, for your detailed analysis on this!
Best regards, Sander.
(Replied this message to Michael's e-mail address, by accident, last week, re-sending to the group now to put proper closure to my question.)
Sander Verhagen
[ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
On 26/12/2018 15:41, Michael Osipov wrote:
I have fixed this already in Wagon 3.2.0 in WAGON-486. Use a Maven
version which has this builtin. It should fix your NAT problem.
Reread the tickets, my explanations were very detailed.
Michael
Am 2018-12-27 um 00:20 schrieb Sander Verhagen:
Hi all,
I'm sorry for reposting this[1]. But I'm sure there is someone in this community who could inspire me to fix this. We are still seeing a lot of stalled Maven builds on Bitbucket, and we don't know anymore what to do about it, besides moving off of Bitbucket (which we've talked about... red tape). Your help would be much appreciated!!!
Best regards, Sander.
[1] https://mail-archives.apache.org/mod_mbox/maven-users/201812.mbox/%3C8ccd358932e3475a91243cb785a15ce5%40MBX14D-ORD1.mex06.mlsrvr.com%3E
Sander Verhagen
[ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
On 11/12/2018 13:19, Sander Verhagen wrote:
Hi all,
Need some help! My current gig is running its Maven builds on Bitbucket. On a regular basis, builds stall while downloading dependencies (I'd say 5%~10% of all builds). It looks a lot like [1]. The proposed solution (maven.wagon.http.pool=false) does not show any noticeable improvement. Atlassian declined to investigate, even though this only ever happens on Bitbucket (local builds have never been seen having this problem).
I have included some logging (-X) from just before the stall. The stalls can happen anywhere dependencies are downloaded, so seemingly unrelated to this particular goal.
I would love to get some suggestions on how to fix this, or at least on how to troubleshoot it further.
Thank you!
[1] https://bitbucket.org/site/master/issues/13988/pipelines-kills-idle-maven-connections
Sander Verhagen
[ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
[DEBUG] -----------------------------------------------------------------------
[DEBUG] Goal: com.github.warmuuh:pactbroker-maven-plugin:0.0.13:upload-pacts (upload-pacts)
[DEBUG] Style: Regular
[DEBUG] Configuration: <?xml version="1.0" encoding="UTF-8"?>
<configuration>
<... snip ...>
</configuration>
[DEBUG] =======================================================================
[DEBUG] Could not find metadata commons-io:commons-io/maven-metadata.xml in local (/root/.m2/repository)
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://jitpack.io
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://repo.spring.io/libs-snapshot
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://jitpack.io
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
[INFO] Downloading from some_repo_snapshot: http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/commons-io/commons-io/maven-metadata.xml
[INFO] Downloading from jitpack.io: https://jitpack.io/commons-io/commons-io/maven-metadata.xml
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://repo.spring.io/libs-snapshot
[INFO] Downloading from spring-snapshots: http://repo.spring.io/libs-snapshot/commons-io/commons-io/maven-metadata.xml
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
[INFO] Downloading from some_repo_release: http://some_repo_release/nexus/content/repositories/some_repo_release/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://oss.sonatype.org/content/repositories/snapshots
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://oss.sonatype.org/content/repositories/snapshots
[INFO] Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://clojars.org/repo
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://clojars.org/repo
[INFO] Downloading from clojars: http://clojars.org/repo/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
// nothing new happened here for 20 minutes until Bitbucket (rightfully) killed the build
Re: Maven builds stalling on Bitbucket
Posted by Michael Osipov <mi...@apache.org>.
I have fixed this already in Wagon 3.2.0 in WAGON-486. Use a Maven
version which has this builtin. It should fix your NAT problem.
Reread the tickets, my explanations were very detailed.
Michael
Am 2018-12-27 um 00:20 schrieb Sander Verhagen:
> Hi all,
>
>
> I'm sorry for reposting this[1]. But I'm sure there is someone in this community who could inspire me to fix this. We are still seeing a lot of stalled Maven builds on Bitbucket, and we don't know anymore what to do about it, besides moving off of Bitbucket (which we've talked about... red tape). Your help would be much appreciated!!!
>
> Best regards, Sander.
>
> [1] https://mail-archives.apache.org/mod_mbox/maven-users/201812.mbox/%3C8ccd358932e3475a91243cb785a15ce5%40MBX14D-ORD1.mex06.mlsrvr.com%3E
>
>
> Sander Verhagen
> [ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
>
> On 11/12/2018 13:19, Sander Verhagen wrote:
>
> Hi all,
>
>
> Need some help! My current gig is running its Maven builds on Bitbucket. On a regular basis, builds stall while downloading dependencies (I'd say 5%~10% of all builds). It looks a lot like [1]. The proposed solution (maven.wagon.http.pool=false) does not show any noticeable improvement. Atlassian declined to investigate, even though this only ever happens on Bitbucket (local builds have never been seen having this problem).
>
> I have included some logging (-X) from just before the stall. The stalls can happen anywhere dependencies are downloaded, so seemingly unrelated to this particular goal.
>
> I would love to get some suggestions on how to fix this, or at least on how to troubleshoot it further.
>
> Thank you!
>
> [1] https://bitbucket.org/site/master/issues/13988/pipelines-kills-idle-maven-connections
>
>
> Sander Verhagen
> [ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
>
>
>
> [DEBUG] -----------------------------------------------------------------------
> [DEBUG] Goal: com.github.warmuuh:pactbroker-maven-plugin:0.0.13:upload-pacts (upload-pacts)
> [DEBUG] Style: Regular
> [DEBUG] Configuration: <?xml version="1.0" encoding="UTF-8"?>
> <configuration>
> <... snip ...>
> </configuration>
> [DEBUG] =======================================================================
> [DEBUG] Could not find metadata commons-io:commons-io/maven-metadata.xml in local (/root/.m2/repository)
> [DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
> [DEBUG] Using transporter WagonTransporter with priority -1.0 for https://jitpack.io
> [DEBUG] Using transporter WagonTransporter with priority -1.0 for http://repo.spring.io/libs-snapshot
> [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://jitpack.io
> [DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
> [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
> [INFO] Downloading from some_repo_snapshot: http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/commons-io/commons-io/maven-metadata.xml
> [INFO] Downloading from jitpack.io: https://jitpack.io/commons-io/commons-io/maven-metadata.xml
> [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://repo.spring.io/libs-snapshot
> [INFO] Downloading from spring-snapshots: http://repo.spring.io/libs-snapshot/commons-io/commons-io/maven-metadata.xml
> [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
> [INFO] Downloading from some_repo_release: http://some_repo_release/nexus/content/repositories/some_repo_release/commons-io/commons-io/maven-metadata.xml
> [DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
> [DEBUG] Using transporter WagonTransporter with priority -1.0 for https://oss.sonatype.org/content/repositories/snapshots
> [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://oss.sonatype.org/content/repositories/snapshots
> [INFO] Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-io/commons-io/maven-metadata.xml
> [DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
> [DEBUG] Using transporter WagonTransporter with priority -1.0 for http://clojars.org/repo
> [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://clojars.org/repo
> [INFO] Downloading from clojars: http://clojars.org/repo/commons-io/commons-io/maven-metadata.xml
> [DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
> [DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2
> [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
> [INFO] Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/maven-metadata.xml
> [DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
> [DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
> // nothing new happened here for 20 minutes until Bitbucket (rightfully) killed the build
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Maven builds stalling on Bitbucket
Posted by Sander Verhagen <sa...@sanderverhagen.net>.
Hi all,
I'm sorry for reposting this[1]. But I'm sure there is someone in this community who could inspire me to fix this. We are still seeing a lot of stalled Maven builds on Bitbucket, and we don't know anymore what to do about it, besides moving off of Bitbucket (which we've talked about... red tape). Your help would be much appreciated!!!
Best regards, Sander.
[1] https://mail-archives.apache.org/mod_mbox/maven-users/201812.mbox/%3C8ccd358932e3475a91243cb785a15ce5%40MBX14D-ORD1.mex06.mlsrvr.com%3E
Sander Verhagen
[ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
On 11/12/2018 13:19, Sander Verhagen wrote:
Hi all,
Need some help! My current gig is running its Maven builds on Bitbucket. On a regular basis, builds stall while downloading dependencies (I'd say 5%~10% of all builds). It looks a lot like [1]. The proposed solution (maven.wagon.http.pool=false) does not show any noticeable improvement. Atlassian declined to investigate, even though this only ever happens on Bitbucket (local builds have never been seen having this problem).
I have included some logging (-X) from just before the stall. The stalls can happen anywhere dependencies are downloaded, so seemingly unrelated to this particular goal.
I would love to get some suggestions on how to fix this, or at least on how to troubleshoot it further.
Thank you!
[1] https://bitbucket.org/site/master/issues/13988/pipelines-kills-idle-maven-connections
Sander Verhagen
[ sander@sanderverhagen.net<ma...@sanderverhagen.net> ]
[DEBUG] -----------------------------------------------------------------------
[DEBUG] Goal: com.github.warmuuh:pactbroker-maven-plugin:0.0.13:upload-pacts (upload-pacts)
[DEBUG] Style: Regular
[DEBUG] Configuration: <?xml version="1.0" encoding="UTF-8"?>
<configuration>
<... snip ...>
</configuration>
[DEBUG] =======================================================================
[DEBUG] Could not find metadata commons-io:commons-io/maven-metadata.xml in local (/root/.m2/repository)
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://jitpack.io
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://repo.spring.io/libs-snapshot
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://jitpack.io
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/
[INFO] Downloading from some_repo_snapshot: http://some_repo_snapshot/nexus/content/repositories/some_repo_snapshot/commons-io/commons-io/maven-metadata.xml
[INFO] Downloading from jitpack.io: https://jitpack.io/commons-io/commons-io/maven-metadata.xml
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://repo.spring.io/libs-snapshot
[INFO] Downloading from spring-snapshots: http://repo.spring.io/libs-snapshot/commons-io/commons-io/maven-metadata.xml
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://some_repo_release/nexus/content/repositories/some_repo_release/
[INFO] Downloading from some_repo_release: http://some_repo_release/nexus/content/repositories/some_repo_release/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://oss.sonatype.org/content/repositories/snapshots
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://oss.sonatype.org/content/repositories/snapshots
[INFO] Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for http://clojars.org/repo
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for http://clojars.org/repo
[INFO] Downloading from clojars: http://clojars.org/repo/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2
[DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/maven-metadata.xml
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
[DEBUG] Writing tracking file /root/.m2/repository/commons-io/commons-io/resolver-status.properties
// nothing new happened here for 20 minutes until Bitbucket (rightfully) killed the build