You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ftpserver-users@mina.apache.org by Kenneth Vanvik Hansen <kv...@online.no> on 2010/05/03 12:37:08 UTC
RE: LIST from database
Hi, i've implemented your new code. I still have some trouble with it being
very slow. Listing 12 directories takes about 100 seconds. This is from a
bit complicated view, which might be some of the reason. Still shouldn't
take that long? Any ideas how to make it even faster?
Kenneth
-----Original Message-----
From: Kenneth Vanvik Hansen [mailto:kvhansen@online.no]
Sent: 27. april 2010 23:30
To: ftpserver-users@mina.apache.org
Subject: Re: LIST from database
Niice :-) I'll take a look at it tomorrow. I wrote a db view that creates a
folder structure form existing blobs. And there are several thousand blobs,
which is how i found the list command being slow. Thank you.
Kenneth
On Apr 27, 2010, at 11:13 PM, Brett M. Bergquist wrote:
> I just did a quick modification of my code to not do a select on each
file. Basically when the "list" is done, I gather all of the information and
cache it in the DbFile object. This greatly increased performance so that
at directory with a couple of hundred files takes less than a second to
return a directory listing.
>
> I have created a Jira issue and attached a Netbeans project which is a
running sample against a Derby database. This has this changed code in it.
So you might want to download the code and make the same changes in your
version.
>
> This is Jira Issue: FTPSERVER-372 </jira/browse/FTPSERVER-372>
>
> https://issues.apache.org/jira/browse/FTPSERVER-372
>
>
> Brett M. Bergquist wrote:
>> Sorry, I just saw this among the hundreds of emails that I get ;)
>>
>> I don't have a quick answer. It is doing 1 select to get the list of
files and then for each other check on each file (isDirectory, isReadable,
etc.) it is doing another select. It is a very naive implementation in that
it is caching nothing and always refreshing its knowledge from the database.
And your right, with many files in a directory it is slow and resource
intensive.
>>
>> Kenneth Vanvik Hansen wrote:
>>> No, i don't have a need to do a select for every file. It just seems
>>> it does a lot of queries when it receives a LIST command. Only
>>> change I have done to the original code is I have changed from Derby
>>> to ODBC. It works great for folders with up to 10 or so files in it,
>>> after that it starts to get really slow.
>>>
>>> -----Original Message-----
>>> From: David Latorre [mailto:dvlato@gmail.com] Sent: 27. april 2010
>>> 14:10
>>> To: ftpserver-users
>>> Subject: Re: LIST from database
>>>
>>> If I inderstood you correctly, it is your code that is doing a
>>> SELECT for every file, isn't it?
>>>
>>> Since I haven't seen your code, I cannot give you advice on this,
>>> why do you need to do a select for every file?
>>>
>>> Maybe Brett's solution can help you... did you get the time to work
>>> on this, Brett?
>>>
>>>
>>>
>>>
>>> 2010/4/27 Kenneth Vanvik Hansen <kv...@online.no>:
>>>
>>>> Hi, I've been working on this a bit now but I'm having some trouble
>>>> when there are many files/folders in a folder. Seems like LIST does
>>>> a select for every file in a folder. Any good ideas on how to solve
>>>> this? In my db(which is a bit slow) it can take several minutes to
>>>> list
>>>>
>>> 40+ files/folders.
>>>
>>>> Kenneth
>>>>
>>>> -----Original Message-----
>>>> From: Brett M. Bergquist [mailto:brett@thebergquistfamily.com]
>>>> Sent: 23. mars 2010 17:40
>>>> To: ftpserver-users@mina.apache.org
>>>> Subject: Re: LIST from database
>>>>
>>>> Okay. That will work. Thanks.
>>>>
>>>> Niklas Gustavsson wrote:
>>>>
>>>>> On Tue, Mar 23, 2010 at 4:08 PM, Brett M. Bergquist
<br...@thebergquistfamily.com> wrote:
>>>>>
>>>>>
>>>>>> I actually have a working Netbeans project that works against a
>>>>>> Derby database. I am going to do a little more cleanup and see
>>>>>> if I cannot factor out the database specifics like the db user
>>>>>> manager does and then I would like to provide the whole thing,
>>>>>> probably to go into the examples. So should I send this to you
>>>>>> Niklas when I am
>>>>>>
>>> ready?
>>>
>>>>> If you would like to contribute the code to the project, the best way
is to create a JIRA issue and attach the code there (making sure you check
the donate to ASF checkbox).
>>>>>
>>>>> /niklas
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>
>
Kenneth Vanvik Hansen
kvhansen@online.no
Re: LIST from database
Posted by "Brett M. Bergquist" <br...@thebergquistfamily.com>.
If you find something in the implementation that needs to be changed,
let me know if you could.
Thanks.
Kenneth Vanvik Hansen wrote:
> I think i actually found something now. Need to rewrite some stuff to make
> it work with the views again. Thanks for your help anyway :-)
>
> Kenneth
>
> -----Original Message-----
> From: Brett Bergquist [mailto:brett@thebergquistfamily.com]
> Sent: 3. mai 2010 12:50
> To: ftpserver-users@mina.apache.org
> Subject: Re: LIST from database
>
> Actually right now, no. Maybe throw some debug statements around or profile
> it to see where it is spending the time. Maybe that will give clue as to
> where it needs to be optimized.
>
> Brett
>
> On May 3, 2010, at 6:37 AM, Kenneth Vanvik Hansen wrote:
>
>
>> Hi, i've implemented your new code. I still have some trouble with it
>> being very slow. Listing 12 directories takes about 100 seconds. This
>> is from a bit complicated view, which might be some of the reason.
>> Still shouldn't take that long? Any ideas how to make it even faster?
>>
>> Kenneth
>>
>> -----Original Message-----
>> From: Kenneth Vanvik Hansen [mailto:kvhansen@online.no]
>> Sent: 27. april 2010 23:30
>> To: ftpserver-users@mina.apache.org
>> Subject: Re: LIST from database
>>
>> Niice :-) I'll take a look at it tomorrow. I wrote a db view that
>> creates a folder structure form existing blobs. And there are several
>> thousand blobs, which is how i found the list command being slow. Thank
>>
> you.
>
>> Kenneth
>>
>> On Apr 27, 2010, at 11:13 PM, Brett M. Bergquist wrote:
>>
>>
>>> I just did a quick modification of my code to not do a select on each
>>>
>> file. Basically when the "list" is done, I gather all of the
>> information and cache it in the DbFile object. This greatly increased
>> performance so that at directory with a couple of hundred files takes
>> less than a second to return a directory listing.
>>
>>> I have created a Jira issue and attached a Netbeans project which is
>>> a
>>>
>> running sample against a Derby database. This has this changed code in
>>
> it.
>
>> So you might want to download the code and make the same changes in
>> your version.
>>
>>> This is Jira Issue: FTPSERVER-372 </jira/browse/FTPSERVER-372>
>>>
>>> https://issues.apache.org/jira/browse/FTPSERVER-372
>>>
>>>
>>> Brett M. Bergquist wrote:
>>>
>>>> Sorry, I just saw this among the hundreds of emails that I get ;)
>>>>
>>>> I don't have a quick answer. It is doing 1 select to get the list
>>>> of
>>>>
>> files and then for each other check on each file (isDirectory,
>> isReadable,
>> etc.) it is doing another select. It is a very naive implementation
>> in that it is caching nothing and always refreshing its knowledge from the
>>
> database.
>
>> And your right, with many files in a directory it is slow and resource
>> intensive.
>>
>>>> Kenneth Vanvik Hansen wrote:
>>>>
>>>>> No, i don't have a need to do a select for every file. It just
>>>>> seems it does a lot of queries when it receives a LIST command.
>>>>> Only change I have done to the original code is I have changed from
>>>>> Derby to ODBC. It works great for folders with up to 10 or so files
>>>>> in it, after that it starts to get really slow.
>>>>>
>>>>> -----Original Message-----
>>>>> From: David Latorre [mailto:dvlato@gmail.com] Sent: 27. april 2010
>>>>> 14:10
>>>>> To: ftpserver-users
>>>>> Subject: Re: LIST from database
>>>>>
>>>>> If I inderstood you correctly, it is your code that is doing a
>>>>> SELECT for every file, isn't it?
>>>>>
>>>>> Since I haven't seen your code, I cannot give you advice on this,
>>>>> why do you need to do a select for every file?
>>>>>
>>>>> Maybe Brett's solution can help you... did you get the time to work
>>>>> on this, Brett?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2010/4/27 Kenneth Vanvik Hansen <kv...@online.no>:
>>>>>
>>>>>
>>>>>> Hi, I've been working on this a bit now but I'm having some
>>>>>> trouble when there are many files/folders in a folder. Seems like
>>>>>> LIST does a select for every file in a folder. Any good ideas on
>>>>>> how to solve this? In my db(which is a bit slow) it can take
>>>>>> several minutes to list
>>>>>>
>>>>>>
>>>>> 40+ files/folders.
>>>>>
>>>>>
>>>>>> Kenneth
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Brett M. Bergquist [mailto:brett@thebergquistfamily.com]
>>>>>> Sent: 23. mars 2010 17:40
>>>>>> To: ftpserver-users@mina.apache.org
>>>>>> Subject: Re: LIST from database
>>>>>>
>>>>>> Okay. That will work. Thanks.
>>>>>>
>>>>>> Niklas Gustavsson wrote:
>>>>>>
>>>>>>
>>>>>>> On Tue, Mar 23, 2010 at 4:08 PM, Brett M. Bergquist
>>>>>>>
>> <br...@thebergquistfamily.com> wrote:
>>
>>>>>>>
>>>>>>>> I actually have a working Netbeans project that works against a
>>>>>>>> Derby database. I am going to do a little more cleanup and see
>>>>>>>> if I cannot factor out the database specifics like the db user
>>>>>>>> manager does and then I would like to provide the whole thing,
>>>>>>>> probably to go into the examples. So should I send this to you
>>>>>>>> Niklas when I am
>>>>>>>>
>>>>>>>>
>>>>> ready?
>>>>>
>>>>>
>>>>>>> If you would like to contribute the code to the project, the best
>>>>>>> way
>>>>>>>
>> is to create a JIRA issue and attach the code there (making sure you
>> check the donate to ASF checkbox).
>>
>>>>>>> /niklas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>> Kenneth Vanvik Hansen
>> kvhansen@online.no
>>
>>
>>
>>
>>
>
>
>
>
RE: LIST from database
Posted by Kenneth Vanvik Hansen <kv...@online.no>.
I think i actually found something now. Need to rewrite some stuff to make
it work with the views again. Thanks for your help anyway :-)
Kenneth
-----Original Message-----
From: Brett Bergquist [mailto:brett@thebergquistfamily.com]
Sent: 3. mai 2010 12:50
To: ftpserver-users@mina.apache.org
Subject: Re: LIST from database
Actually right now, no. Maybe throw some debug statements around or profile
it to see where it is spending the time. Maybe that will give clue as to
where it needs to be optimized.
Brett
On May 3, 2010, at 6:37 AM, Kenneth Vanvik Hansen wrote:
> Hi, i've implemented your new code. I still have some trouble with it
> being very slow. Listing 12 directories takes about 100 seconds. This
> is from a bit complicated view, which might be some of the reason.
> Still shouldn't take that long? Any ideas how to make it even faster?
>
> Kenneth
>
> -----Original Message-----
> From: Kenneth Vanvik Hansen [mailto:kvhansen@online.no]
> Sent: 27. april 2010 23:30
> To: ftpserver-users@mina.apache.org
> Subject: Re: LIST from database
>
> Niice :-) I'll take a look at it tomorrow. I wrote a db view that
> creates a folder structure form existing blobs. And there are several
> thousand blobs, which is how i found the list command being slow. Thank
you.
>
> Kenneth
>
> On Apr 27, 2010, at 11:13 PM, Brett M. Bergquist wrote:
>
>> I just did a quick modification of my code to not do a select on each
> file. Basically when the "list" is done, I gather all of the
> information and cache it in the DbFile object. This greatly increased
> performance so that at directory with a couple of hundred files takes
> less than a second to return a directory listing.
>>
>> I have created a Jira issue and attached a Netbeans project which is
>> a
> running sample against a Derby database. This has this changed code in
it.
> So you might want to download the code and make the same changes in
> your version.
>>
>> This is Jira Issue: FTPSERVER-372 </jira/browse/FTPSERVER-372>
>>
>> https://issues.apache.org/jira/browse/FTPSERVER-372
>>
>>
>> Brett M. Bergquist wrote:
>>> Sorry, I just saw this among the hundreds of emails that I get ;)
>>>
>>> I don't have a quick answer. It is doing 1 select to get the list
>>> of
> files and then for each other check on each file (isDirectory,
> isReadable,
> etc.) it is doing another select. It is a very naive implementation
> in that it is caching nothing and always refreshing its knowledge from the
database.
> And your right, with many files in a directory it is slow and resource
> intensive.
>>>
>>> Kenneth Vanvik Hansen wrote:
>>>> No, i don't have a need to do a select for every file. It just
>>>> seems it does a lot of queries when it receives a LIST command.
>>>> Only change I have done to the original code is I have changed from
>>>> Derby to ODBC. It works great for folders with up to 10 or so files
>>>> in it, after that it starts to get really slow.
>>>>
>>>> -----Original Message-----
>>>> From: David Latorre [mailto:dvlato@gmail.com] Sent: 27. april 2010
>>>> 14:10
>>>> To: ftpserver-users
>>>> Subject: Re: LIST from database
>>>>
>>>> If I inderstood you correctly, it is your code that is doing a
>>>> SELECT for every file, isn't it?
>>>>
>>>> Since I haven't seen your code, I cannot give you advice on this,
>>>> why do you need to do a select for every file?
>>>>
>>>> Maybe Brett's solution can help you... did you get the time to work
>>>> on this, Brett?
>>>>
>>>>
>>>>
>>>>
>>>> 2010/4/27 Kenneth Vanvik Hansen <kv...@online.no>:
>>>>
>>>>> Hi, I've been working on this a bit now but I'm having some
>>>>> trouble when there are many files/folders in a folder. Seems like
>>>>> LIST does a select for every file in a folder. Any good ideas on
>>>>> how to solve this? In my db(which is a bit slow) it can take
>>>>> several minutes to list
>>>>>
>>>> 40+ files/folders.
>>>>
>>>>> Kenneth
>>>>>
>>>>> -----Original Message-----
>>>>> From: Brett M. Bergquist [mailto:brett@thebergquistfamily.com]
>>>>> Sent: 23. mars 2010 17:40
>>>>> To: ftpserver-users@mina.apache.org
>>>>> Subject: Re: LIST from database
>>>>>
>>>>> Okay. That will work. Thanks.
>>>>>
>>>>> Niklas Gustavsson wrote:
>>>>>
>>>>>> On Tue, Mar 23, 2010 at 4:08 PM, Brett M. Bergquist
> <br...@thebergquistfamily.com> wrote:
>>>>>>
>>>>>>
>>>>>>> I actually have a working Netbeans project that works against a
>>>>>>> Derby database. I am going to do a little more cleanup and see
>>>>>>> if I cannot factor out the database specifics like the db user
>>>>>>> manager does and then I would like to provide the whole thing,
>>>>>>> probably to go into the examples. So should I send this to you
>>>>>>> Niklas when I am
>>>>>>>
>>>> ready?
>>>>
>>>>>> If you would like to contribute the code to the project, the best
>>>>>> way
> is to create a JIRA issue and attach the code there (making sure you
> check the donate to ASF checkbox).
>>>>>>
>>>>>> /niklas
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
> Kenneth Vanvik Hansen
> kvhansen@online.no
>
>
>
>
Re: LIST from database
Posted by Brett Bergquist <br...@thebergquistfamily.com>.
Actually right now, no. Maybe throw some debug statements around or profile it to see where it is spending the time. Maybe that will give clue as to where it needs to be optimized.
Brett
On May 3, 2010, at 6:37 AM, Kenneth Vanvik Hansen wrote:
> Hi, i've implemented your new code. I still have some trouble with it being
> very slow. Listing 12 directories takes about 100 seconds. This is from a
> bit complicated view, which might be some of the reason. Still shouldn't
> take that long? Any ideas how to make it even faster?
>
> Kenneth
>
> -----Original Message-----
> From: Kenneth Vanvik Hansen [mailto:kvhansen@online.no]
> Sent: 27. april 2010 23:30
> To: ftpserver-users@mina.apache.org
> Subject: Re: LIST from database
>
> Niice :-) I'll take a look at it tomorrow. I wrote a db view that creates a
> folder structure form existing blobs. And there are several thousand blobs,
> which is how i found the list command being slow. Thank you.
>
> Kenneth
>
> On Apr 27, 2010, at 11:13 PM, Brett M. Bergquist wrote:
>
>> I just did a quick modification of my code to not do a select on each
> file. Basically when the "list" is done, I gather all of the information and
> cache it in the DbFile object. This greatly increased performance so that
> at directory with a couple of hundred files takes less than a second to
> return a directory listing.
>>
>> I have created a Jira issue and attached a Netbeans project which is a
> running sample against a Derby database. This has this changed code in it.
> So you might want to download the code and make the same changes in your
> version.
>>
>> This is Jira Issue: FTPSERVER-372 </jira/browse/FTPSERVER-372>
>>
>> https://issues.apache.org/jira/browse/FTPSERVER-372
>>
>>
>> Brett M. Bergquist wrote:
>>> Sorry, I just saw this among the hundreds of emails that I get ;)
>>>
>>> I don't have a quick answer. It is doing 1 select to get the list of
> files and then for each other check on each file (isDirectory, isReadable,
> etc.) it is doing another select. It is a very naive implementation in that
> it is caching nothing and always refreshing its knowledge from the database.
> And your right, with many files in a directory it is slow and resource
> intensive.
>>>
>>> Kenneth Vanvik Hansen wrote:
>>>> No, i don't have a need to do a select for every file. It just seems
>>>> it does a lot of queries when it receives a LIST command. Only
>>>> change I have done to the original code is I have changed from Derby
>>>> to ODBC. It works great for folders with up to 10 or so files in it,
>>>> after that it starts to get really slow.
>>>>
>>>> -----Original Message-----
>>>> From: David Latorre [mailto:dvlato@gmail.com] Sent: 27. april 2010
>>>> 14:10
>>>> To: ftpserver-users
>>>> Subject: Re: LIST from database
>>>>
>>>> If I inderstood you correctly, it is your code that is doing a
>>>> SELECT for every file, isn't it?
>>>>
>>>> Since I haven't seen your code, I cannot give you advice on this,
>>>> why do you need to do a select for every file?
>>>>
>>>> Maybe Brett's solution can help you... did you get the time to work
>>>> on this, Brett?
>>>>
>>>>
>>>>
>>>>
>>>> 2010/4/27 Kenneth Vanvik Hansen <kv...@online.no>:
>>>>
>>>>> Hi, I've been working on this a bit now but I'm having some trouble
>>>>> when there are many files/folders in a folder. Seems like LIST does
>>>>> a select for every file in a folder. Any good ideas on how to solve
>>>>> this? In my db(which is a bit slow) it can take several minutes to
>>>>> list
>>>>>
>>>> 40+ files/folders.
>>>>
>>>>> Kenneth
>>>>>
>>>>> -----Original Message-----
>>>>> From: Brett M. Bergquist [mailto:brett@thebergquistfamily.com]
>>>>> Sent: 23. mars 2010 17:40
>>>>> To: ftpserver-users@mina.apache.org
>>>>> Subject: Re: LIST from database
>>>>>
>>>>> Okay. That will work. Thanks.
>>>>>
>>>>> Niklas Gustavsson wrote:
>>>>>
>>>>>> On Tue, Mar 23, 2010 at 4:08 PM, Brett M. Bergquist
> <br...@thebergquistfamily.com> wrote:
>>>>>>
>>>>>>
>>>>>>> I actually have a working Netbeans project that works against a
>>>>>>> Derby database. I am going to do a little more cleanup and see
>>>>>>> if I cannot factor out the database specifics like the db user
>>>>>>> manager does and then I would like to provide the whole thing,
>>>>>>> probably to go into the examples. So should I send this to you
>>>>>>> Niklas when I am
>>>>>>>
>>>> ready?
>>>>
>>>>>> If you would like to contribute the code to the project, the best way
> is to create a JIRA issue and attach the code there (making sure you check
> the donate to ASF checkbox).
>>>>>>
>>>>>> /niklas
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>
> Kenneth Vanvik Hansen
> kvhansen@online.no
>
>
>
>