You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ant.apache.org by "Johannes Rudolph (JIRA)" <ji...@apache.org> on 2008/11/19 11:09:44 UTC
[jira] Created: (IVY-973) [PATCH] SFTP repositories become very
slow after a while
[PATCH] SFTP repositories become very slow after a while
--------------------------------------------------------
Key: IVY-973
URL: https://issues.apache.org/jira/browse/IVY-973
Project: Ivy
Issue Type: Improvement
Components: Core
Affects Versions: 2.0-RC2
Reporter: Johannes Rudolph
Attachments: faster-sftp.patch
If many versions of a module live in a folder of an SFTP repository each resolve for dynamic versions becomes dreadfully slow because a roundtrip to the server is made to the server for every version which lies on the server.
The call sequence goes like that:
org.apache.ivy.plugins.resolver.util.ResolverHelper.findAll calls
1. org.apache.ivy.plugins.resolver.util.ResolverHelper.listTokenValues which calls
SFTPRepository.list with the directory of the module as parameter (1. roundtrip)
2. for every found file matching the pattern SFTPRepository.getResource(path).exists()
In turn SFTPResource tries to initialize itself and calls SFTPRepository.resolveResource which does the second roundtrip to the server - for every version of the file
Example: Having 20 versions of a module in a repository which is a roundtrip of 150ms away, leads to a lag of 20 * 150ms = 3s . Multiply that with any module having a dynamic version.
The solution is to cache all the resources in SFTPRepository. You can use the information retrieved when calling SFTPRepository.list to fill the cache in advance.
The disadvantage is that you may have stale data. IMO this is mostly no problem because every call to list updates the cache and files already available in the repository won't hopefully change any metadata in the time between.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-973) [PATCH] SFTP repositories become very
slow after a while
Posted by "Maarten Coene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649483#action_12649483 ]
Maarten Coene commented on IVY-973:
-----------------------------------
I think your patch will only work if you don't have a [revision] token in your directory structure.
Can you confirm this is the case in your situation?
So if your pattern looks like for instance "[organisation]/[module]/[revision]/[artifact]-[revision].[ext]", you'll still have a roundtrip for each revision.
I think we should try to avoid calling Resource.exists() in the ResolverHelper.findAll() method.
> [PATCH] SFTP repositories become very slow after a while
> --------------------------------------------------------
>
> Key: IVY-973
> URL: https://issues.apache.org/jira/browse/IVY-973
> Project: Ivy
> Issue Type: Improvement
> Components: Core
> Affects Versions: 2.0-RC2
> Reporter: Johannes Rudolph
> Priority: Minor
> Attachments: faster-sftp.patch
>
>
> If many versions of a module live in a folder of an SFTP repository each resolve for dynamic versions becomes dreadfully slow because a roundtrip to the server is made to the server for every version which lies on the server.
> The call sequence goes like that:
> org.apache.ivy.plugins.resolver.util.ResolverHelper.findAll calls
> 1. org.apache.ivy.plugins.resolver.util.ResolverHelper.listTokenValues which calls
> SFTPRepository.list with the directory of the module as parameter (1. roundtrip)
> 2. for every found file matching the pattern SFTPRepository.getResource(path).exists()
> In turn SFTPResource tries to initialize itself and calls SFTPRepository.resolveResource which does the second roundtrip to the server - for every version of the file
> Example: Having 20 versions of a module in a repository which is a roundtrip of 150ms away, leads to a lag of 20 * 150ms = 3s . Multiply that with any module having a dynamic version.
> The solution is to cache all the resources in SFTPRepository. You can use the information retrieved when calling SFTPRepository.list to fill the cache in advance.
> The disadvantage is that you may have stale data. IMO this is mostly no problem because every call to list updates the cache and files already available in the repository won't hopefully change any metadata in the time between.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-973) [PATCH] SFTP repositories become very
slow after a while
Posted by "Johannes Rudolph (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johannes Rudolph updated IVY-973:
---------------------------------
Attachment: faster-sftp.patch
A patch which fixes the problem by introducing a cache.
> [PATCH] SFTP repositories become very slow after a while
> --------------------------------------------------------
>
> Key: IVY-973
> URL: https://issues.apache.org/jira/browse/IVY-973
> Project: Ivy
> Issue Type: Improvement
> Components: Core
> Affects Versions: 2.0-RC2
> Reporter: Johannes Rudolph
> Attachments: faster-sftp.patch
>
>
> If many versions of a module live in a folder of an SFTP repository each resolve for dynamic versions becomes dreadfully slow because a roundtrip to the server is made to the server for every version which lies on the server.
> The call sequence goes like that:
> org.apache.ivy.plugins.resolver.util.ResolverHelper.findAll calls
> 1. org.apache.ivy.plugins.resolver.util.ResolverHelper.listTokenValues which calls
> SFTPRepository.list with the directory of the module as parameter (1. roundtrip)
> 2. for every found file matching the pattern SFTPRepository.getResource(path).exists()
> In turn SFTPResource tries to initialize itself and calls SFTPRepository.resolveResource which does the second roundtrip to the server - for every version of the file
> Example: Having 20 versions of a module in a repository which is a roundtrip of 150ms away, leads to a lag of 20 * 150ms = 3s . Multiply that with any module having a dynamic version.
> The solution is to cache all the resources in SFTPRepository. You can use the information retrieved when calling SFTPRepository.list to fill the cache in advance.
> The disadvantage is that you may have stale data. IMO this is mostly no problem because every call to list updates the cache and files already available in the repository won't hopefully change any metadata in the time between.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (IVY-973) [PATCH] SFTP repositories become very
slow after a while
Posted by "Johannes Rudolph (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johannes Rudolph updated IVY-973:
---------------------------------
Priority: Minor (was: Major)
> [PATCH] SFTP repositories become very slow after a while
> --------------------------------------------------------
>
> Key: IVY-973
> URL: https://issues.apache.org/jira/browse/IVY-973
> Project: Ivy
> Issue Type: Improvement
> Components: Core
> Affects Versions: 2.0-RC2
> Reporter: Johannes Rudolph
> Priority: Minor
> Attachments: faster-sftp.patch
>
>
> If many versions of a module live in a folder of an SFTP repository each resolve for dynamic versions becomes dreadfully slow because a roundtrip to the server is made to the server for every version which lies on the server.
> The call sequence goes like that:
> org.apache.ivy.plugins.resolver.util.ResolverHelper.findAll calls
> 1. org.apache.ivy.plugins.resolver.util.ResolverHelper.listTokenValues which calls
> SFTPRepository.list with the directory of the module as parameter (1. roundtrip)
> 2. for every found file matching the pattern SFTPRepository.getResource(path).exists()
> In turn SFTPResource tries to initialize itself and calls SFTPRepository.resolveResource which does the second roundtrip to the server - for every version of the file
> Example: Having 20 versions of a module in a repository which is a roundtrip of 150ms away, leads to a lag of 20 * 150ms = 3s . Multiply that with any module having a dynamic version.
> The solution is to cache all the resources in SFTPRepository. You can use the information retrieved when calling SFTPRepository.list to fill the cache in advance.
> The disadvantage is that you may have stale data. IMO this is mostly no problem because every call to list updates the cache and files already available in the repository won't hopefully change any metadata in the time between.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (IVY-973) [PATCH] SFTP repositories become very
slow after a while
Posted by "Johannes Rudolph (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/IVY-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649620#action_12649620 ]
Johannes Rudolph commented on IVY-973:
--------------------------------------
Yes, you are right, we have a much simpler directory structure which is like [organization]/[module]-[artifact]-[revision].[ext].
Removing Resource.exists probably would be a better idea, since a resource that was found before should be available. But that would broaden the scope of this patch since every resolver would be concerned. Is it a safe thing to do?!?
> [PATCH] SFTP repositories become very slow after a while
> --------------------------------------------------------
>
> Key: IVY-973
> URL: https://issues.apache.org/jira/browse/IVY-973
> Project: Ivy
> Issue Type: Improvement
> Components: Core
> Affects Versions: 2.0-RC2
> Reporter: Johannes Rudolph
> Priority: Minor
> Attachments: faster-sftp.patch
>
>
> If many versions of a module live in a folder of an SFTP repository each resolve for dynamic versions becomes dreadfully slow because a roundtrip to the server is made to the server for every version which lies on the server.
> The call sequence goes like that:
> org.apache.ivy.plugins.resolver.util.ResolverHelper.findAll calls
> 1. org.apache.ivy.plugins.resolver.util.ResolverHelper.listTokenValues which calls
> SFTPRepository.list with the directory of the module as parameter (1. roundtrip)
> 2. for every found file matching the pattern SFTPRepository.getResource(path).exists()
> In turn SFTPResource tries to initialize itself and calls SFTPRepository.resolveResource which does the second roundtrip to the server - for every version of the file
> Example: Having 20 versions of a module in a repository which is a roundtrip of 150ms away, leads to a lag of 20 * 150ms = 3s . Multiply that with any module having a dynamic version.
> The solution is to cache all the resources in SFTPRepository. You can use the information retrieved when calling SFTPRepository.list to fill the cache in advance.
> The disadvantage is that you may have stale data. IMO this is mostly no problem because every call to list updates the cache and files already available in the repository won't hopefully change any metadata in the time between.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.