You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Ralph Goers <ra...@dslextreme.com> on 2009/02/23 07:28:00 UTC
[VFS]HttpConnectionManager
I've been working on getting Commons Configuration to be able to use
VFS as its underlying file system and have everything working.
However, in the course of this effort I ran into a problem. https://issues.apache.org/jira/browse/VFS-164
states that there were problems under load and so
ThreadLocalHttpConnectionManager was written and replaced the use of
MultiThreadedHttpConnectionManager.
The issue I ran into is that ThreadLocalHttpConnectionManager only
allows a single connection open on a thread. This causes some fairly
common use cases in Commons Configuration to fail. To get around this
I have gone back to MultiThreadedHttpConnectionManager but provided
mechanisms to configure maxConnectionsPerHost and
maxTotalConnections. Doing this got, along with a few other changes,
got everything to work.
What I'd like to know is, was there more to VFS-164 than is stated in
the issue and is this change sufficient? Or do I need to create yet
another HttpConnectionManager?
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS]HttpConnectionManager
Posted by Ralph Goers <ra...@dslextreme.com>.
On Feb 23, 2009, at 12:18 AM, Mario Ivankovits wrote:
>
>
>> I've added maxConnectionsPerHost and
>> maxTotalConnections to HttpFileSystemConfigBuilder and also allowed
>> them to be specified as system properties.
>
> You would make a lot of people happy (I think) if you implement this
> in a commons way, so that every FileSystemOption can also be
> specified using system properties. Or did you already? :-)
>
> Something like -Dvfsopt.ftp.passiveMode=true
>
> You can use the DelegatingFileSystemOptionsBuilder to help you here.
>
> FileSystemOptions fso = new FileSystemOptions();
> DelegatingFileSystemOptionsBuilder delegate = new
> DelegatingFileSystemOptionsBuilder(VFS.getManager());
> delegate.setConfigString(fso, "sftp", "identities", "c:/tmp/
> test.ident");
> delegate.setConfigString(fso, "http", "proxyPort", "8080");
> delegate.setConfigClass(fso, "sftp", "userinfo",
> TrustEveryoneUserInfo.class);
>
> by iterating over the System properties, taking each vfsopt.* and
> split it so taht you can pass it to setConfigString.
>
> This can do the trick.
>
Thanks. No, I hadn't done that but I will definitely give it a try!
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [VFS]HttpConnectionManager
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi!
> -----Original Message-----
> From: Ralph Goers [mailto:ralph.goers@dslextreme.com]
> Sent: Monday, February 23, 2009 8:29 AM
>
> Thanks Mario. VFS-164 wasn't really clear. Was the problem the limit
> to 2 connections per host that MultiThreadedHttpConnectionManager has
> by default?
Sorry, I can't remember :-( Yes, I think it hang, and I am not sure if the maxConnectionsPerHost existed at this time, or probably I just overlooked it *grmpf*.
> I've added maxConnectionsPerHost and
> maxTotalConnections to HttpFileSystemConfigBuilder and also allowed
> them to be specified as system properties.
You would make a lot of people happy (I think) if you implement this in a commons way, so that every FileSystemOption can also be specified using system properties. Or did you already? :-)
Something like -Dvfsopt.ftp.passiveMode=true
You can use the DelegatingFileSystemOptionsBuilder to help you here.
FileSystemOptions fso = new FileSystemOptions();
DelegatingFileSystemOptionsBuilder delegate = new DelegatingFileSystemOptionsBuilder(VFS.getManager());
delegate.setConfigString(fso, "sftp", "identities", "c:/tmp/test.ident");
delegate.setConfigString(fso, "http", "proxyPort", "8080");
delegate.setConfigClass(fso, "sftp", "userinfo", TrustEveryoneUserInfo.class);
by iterating over the System properties, taking each vfsopt.* and split it so taht you can pass it to setConfigString.
This can do the trick.
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [VFS]HttpConnectionManager
Posted by Ralph Goers <ra...@dslextreme.com>.
Thanks Mario. VFS-164 wasn't really clear. Was the problem the limit
to 2 connections per host that MultiThreadedHttpConnectionManager has
by default? I changed VFS to have it default to 5 and the unit tests
pass - including ContentTests.testConcurrentRead() which would hang
with only 2 connections. I've added maxConnectionsPerHost and
maxTotalConnections to HttpFileSystemConfigBuilder and also allowed
them to be specified as system properties.
Ralph
On Feb 22, 2009, at 11:15 PM, Mario Ivankovits wrote:
> Hi!
>> -----Original Message-----
>> From: Ralph Goers [mailto:ralph.goers@dslextreme.com]
>> Sent: Monday, February 23, 2009 7:28 AM
>>
>> What I'd like to know is, was there more to VFS-164 than is stated in
>> the issue and is this change sufficient? Or do I need to create yet
>> another HttpConnectionManager?
>
> No, it was just VFS-164 and if you found another solution to that
> which allows multiple files now it is good! :-)
>
> Ciao,
> Mario
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [VFS]HttpConnectionManager
Posted by Mario Ivankovits <ma...@ops.co.at>.
Hi!
> -----Original Message-----
> From: Ralph Goers [mailto:ralph.goers@dslextreme.com]
> Sent: Monday, February 23, 2009 7:28 AM
>
> What I'd like to know is, was there more to VFS-164 than is stated in
> the issue and is this change sufficient? Or do I need to create yet
> another HttpConnectionManager?
No, it was just VFS-164 and if you found another solution to that which allows multiple files now it is good! :-)
Ciao,
Mario
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org