You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ant.apache.org by "Klaas Prause (JIRA)" <ji...@apache.org> on 2009/04/29 16:53:30 UTC
[jira] Created: (IVY-1073) publish should be an atomic operation
publish should be an atomic operation
-------------------------------------
Key: IVY-1073
URL: https://issues.apache.org/jira/browse/IVY-1073
Project: Ivy
Issue Type: Improvement
Affects Versions: 2.1.0-RC1, 1.4.1
Reporter: Klaas Prause
The publish action is not atomic, so it does not succeed as one action or fails completly. This has two problems:
1. We are using ivy not only for publishing JARs but also for the releaseables including EARs and WARs. So the publish target sometimes uploads more than 50MB what takes a few seconds. Our continious integration server builds with ~6 build queues. So it often happens that a build is checking for new versions and finds a version that is currently in the process of publishing. The retrieve fails because not all items are uploaded and the build fails.
2. If a build did not produce all needed artifacts (a clear build error) Ivy publish does not tests for missing artifacts prior starting the upload. If an artifact is missing the publish fails but the already uploaded files are not deleted. All builds other builds try to retrieve the incomplete published artifact and fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1073) publish should be an atomic operation
Posted by "Klaas Prause (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704532#action_12704532 ]
Klaas Prause commented on IVY-1073:
-----------------------------------
I tried to use the caches lockStrategy="artifact-lock" option, but with a lock enabled the retrieve does not start work, with an empty cache it stops trying to resolve with no progress at all. If the cache is filled with all artifacts it stops after resolve and does not retrieve the artifacts.
Is this a bug in the lock strategy option?
> publish should be an atomic operation
> -------------------------------------
>
> Key: IVY-1073
> URL: https://issues.apache.org/jira/browse/IVY-1073
> Project: Ivy
> Issue Type: Bug
> Affects Versions: 1.4.1, 2.1.0-RC1
> Reporter: Klaas Prause
> Attachments: ivysettings-resolver.xml
>
>
> The publish action is not atomic, so it does not succeed as one action or fails completly. This has two problems:
> 1. We are using ivy not only for publishing JARs but also for the releaseables including EARs and WARs. So the publish target sometimes uploads more than 50MB what takes a few seconds. Our continious integration server builds with ~6 build queues. So it often happens that a build is checking for new versions and finds a version that is currently in the process of publishing. The retrieve fails because not all items are uploaded and the build fails.
> 2. If a build did not produce all needed artifacts (a clear build error) Ivy publish does not tests for missing artifacts prior starting the upload. If an artifact is missing the publish fails but the already uploaded files are not deleted. All builds other builds try to retrieve the incomplete published artifact and fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1073) publish should be an atomic operation
Posted by "Klaas Prause (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704519#action_12704519 ]
Klaas Prause commented on IVY-1073:
-----------------------------------
Maybe the synchronization could be applied to the local cache as well. On the build machine we are using one ivy cache with many ivy processes acting on it. We get errors where all dependencies are found but the checksums are incorrect. Triggering the same build again fixes the problem. The new build did not retrieve new artifacts, so it seems that the access to the local cache has the same problems as described for publish to the repository.
> publish should be an atomic operation
> -------------------------------------
>
> Key: IVY-1073
> URL: https://issues.apache.org/jira/browse/IVY-1073
> Project: Ivy
> Issue Type: Bug
> Affects Versions: 1.4.1, 2.1.0-RC1
> Reporter: Klaas Prause
>
> The publish action is not atomic, so it does not succeed as one action or fails completly. This has two problems:
> 1. We are using ivy not only for publishing JARs but also for the releaseables including EARs and WARs. So the publish target sometimes uploads more than 50MB what takes a few seconds. Our continious integration server builds with ~6 build queues. So it often happens that a build is checking for new versions and finds a version that is currently in the process of publishing. The retrieve fails because not all items are uploaded and the build fails.
> 2. If a build did not produce all needed artifacts (a clear build error) Ivy publish does not tests for missing artifacts prior starting the upload. If an artifact is missing the publish fails but the already uploaded files are not deleted. All builds other builds try to retrieve the incomplete published artifact and fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1073) publish should be an atomic operation
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704520#action_12704520 ]
Maarten Coene commented on IVY-1073:
------------------------------------
What resolver do you use for publishing? The filesystem resolver has atomic publish support:
http://ant.apache.org/ivy/history/2.1.0-rc1/resolver/filesystem.html
> publish should be an atomic operation
> -------------------------------------
>
> Key: IVY-1073
> URL: https://issues.apache.org/jira/browse/IVY-1073
> Project: Ivy
> Issue Type: Bug
> Affects Versions: 1.4.1, 2.1.0-RC1
> Reporter: Klaas Prause
>
> The publish action is not atomic, so it does not succeed as one action or fails completly. This has two problems:
> 1. We are using ivy not only for publishing JARs but also for the releaseables including EARs and WARs. So the publish target sometimes uploads more than 50MB what takes a few seconds. Our continious integration server builds with ~6 build queues. So it often happens that a build is checking for new versions and finds a version that is currently in the process of publishing. The retrieve fails because not all items are uploaded and the build fails.
> 2. If a build did not produce all needed artifacts (a clear build error) Ivy publish does not tests for missing artifacts prior starting the upload. If an artifact is missing the publish fails but the already uploaded files are not deleted. All builds other builds try to retrieve the incomplete published artifact and fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-1073) publish should be an atomic operation
Posted by "Klaas Prause (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704138#action_12704138 ]
Klaas Prause commented on IVY-1073:
-----------------------------------
maybe ivy could establish some locking mechanism via lock files or rename the directory after the publish is finished and delete it if publishing fails.
> publish should be an atomic operation
> -------------------------------------
>
> Key: IVY-1073
> URL: https://issues.apache.org/jira/browse/IVY-1073
> Project: Ivy
> Issue Type: Improvement
> Affects Versions: 1.4.1, 2.1.0-RC1
> Reporter: Klaas Prause
>
> The publish action is not atomic, so it does not succeed as one action or fails completly. This has two problems:
> 1. We are using ivy not only for publishing JARs but also for the releaseables including EARs and WARs. So the publish target sometimes uploads more than 50MB what takes a few seconds. Our continious integration server builds with ~6 build queues. So it often happens that a build is checking for new versions and finds a version that is currently in the process of publishing. The retrieve fails because not all items are uploaded and the build fails.
> 2. If a build did not produce all needed artifacts (a clear build error) Ivy publish does not tests for missing artifacts prior starting the upload. If an artifact is missing the publish fails but the already uploaded files are not deleted. All builds other builds try to retrieve the incomplete published artifact and fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-1073) publish should be an atomic operation
Posted by "Klaas Prause (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Prause updated IVY-1073:
------------------------------
Attachment: ivysettings-resolver.xml
I attached our resolvers, sadly we have no file system resolvers, because our repository is hosted on a central server. Only the cache is a local filesystem resolver. We are using this pattern "${ivy.dir.pattern}/[revision]/[artifact].[ext]" so we should fullfil the requirements for the atomic operation.
Is it planned to add the atomic feature to all resolvers?
> publish should be an atomic operation
> -------------------------------------
>
> Key: IVY-1073
> URL: https://issues.apache.org/jira/browse/IVY-1073
> Project: Ivy
> Issue Type: Bug
> Affects Versions: 1.4.1, 2.1.0-RC1
> Reporter: Klaas Prause
> Attachments: ivysettings-resolver.xml
>
>
> The publish action is not atomic, so it does not succeed as one action or fails completly. This has two problems:
> 1. We are using ivy not only for publishing JARs but also for the releaseables including EARs and WARs. So the publish target sometimes uploads more than 50MB what takes a few seconds. Our continious integration server builds with ~6 build queues. So it often happens that a build is checking for new versions and finds a version that is currently in the process of publishing. The retrieve fails because not all items are uploaded and the build fails.
> 2. If a build did not produce all needed artifacts (a clear build error) Ivy publish does not tests for missing artifacts prior starting the upload. If an artifact is missing the publish fails but the already uploaded files are not deleted. All builds other builds try to retrieve the incomplete published artifact and fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-1073) publish should be an atomic operation
Posted by "Klaas Prause (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Klaas Prause updated IVY-1073:
------------------------------
Issue Type: Bug (was: Improvement)
switched from improvement to bug:
For our current build process a lot of builds fail because of the missing synchronization between different ivy instances acting on the ivy repository. We need to find a work around to distinguish between real and ivy related build failures.
> publish should be an atomic operation
> -------------------------------------
>
> Key: IVY-1073
> URL: https://issues.apache.org/jira/browse/IVY-1073
> Project: Ivy
> Issue Type: Bug
> Affects Versions: 1.4.1, 2.1.0-RC1
> Reporter: Klaas Prause
>
> The publish action is not atomic, so it does not succeed as one action or fails completly. This has two problems:
> 1. We are using ivy not only for publishing JARs but also for the releaseables including EARs and WARs. So the publish target sometimes uploads more than 50MB what takes a few seconds. Our continious integration server builds with ~6 build queues. So it often happens that a build is checking for new versions and finds a version that is currently in the process of publishing. The retrieve fails because not all items are uploaded and the build fails.
> 2. If a build did not produce all needed artifacts (a clear build error) Ivy publish does not tests for missing artifacts prior starting the upload. If an artifact is missing the publish fails but the already uploaded files are not deleted. All builds other builds try to retrieve the incomplete published artifact and fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.