You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Nicolas (Created) (JIRA)" <ji...@apache.org> on 2012/03/20 16:43:40 UTC
[jira] [Created] (NET-455) FTPClient listFiles(FTPFileFilter) can
be faster...
FTPClient listFiles(FTPFileFilter) can be faster...
---------------------------------------------------
Key: NET-455
URL: https://issues.apache.org/jira/browse/NET-455
Project: Commons Net
Issue Type: Improvement
Components: FTP
Affects Versions: 3.0.1
Environment: all
Reporter: Nicolas
Priority: Minor
I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
The first one is the conventionnal => files = ftpClient.listFiles("./", ftpFileFilter);
It takes about 90 seconds ...
the second one is ... nearly as simple as the first one :
FTPFile [] allFilesListed = ftpClient.listFiles();
files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
for (FTPFile ftpf : ftpFileArray) {
boolean accepted = filter.accept(ftpf);
if (accepted) {
listOfFTPFiles.add(ftpf);
}
}
FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
return listOfFTPFiles.toArray(resultArray);
}
The second method is EIGHT times faster than the first one ... and I don't really understand why...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NET-455) FTPClient listFiles(FTPFileFilter) can
be faster...
Posted by "Sebb (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NET-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236849#comment-13236849 ]
Sebb commented on NET-455:
--------------------------
Does seem rather odd, because the listFiles() method actually filters the files using FTPFileFilters.NON_NULL.
The second method should be slightly slower.
One possibility is that the FTP server (or its OS) does some caching of the file directory listing, so that fetching the second listing is much faster than the first.
Try repeating the test the other way around, or just repeating both methods A & B - e.g. ABAB - in a single test.
If the behaviour really is repeatable, can you provide a simple (complete) test case that shows the behaviour?
> FTPClient listFiles(FTPFileFilter) can be faster...
> ---------------------------------------------------
>
> Key: NET-455
> URL: https://issues.apache.org/jira/browse/NET-455
> Project: Commons Net
> Issue Type: Improvement
> Components: FTP
> Affects Versions: 3.0.1
> Environment: all
> Reporter: Nicolas
> Priority: Minor
> Labels: ftpclient, listFiles
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
> The first one is the conventionnal =>
> {code}
> files = ftpClient.listFiles("./", ftpFileFilter);
> {code}
> It takes about 90 seconds ...
> the second one is ... nearly as simple as the first one :
> {code}
> FTPFile [] allFilesListed = ftpClient.listFiles();
> files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
> {code}
> The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
> {code}
> public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
> ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
> for (FTPFile ftpf : ftpFileArray) {
> boolean accepted = filter.accept(ftpf);
> if (accepted) {
> listOfFTPFiles.add(ftpf);
> }
> }
> FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
> return listOfFTPFiles.toArray(resultArray);
> }
> {code}
> The second method is EIGHT times faster than the first one ... and I don't really understand why...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (NET-455) FTPClient listFiles(FTPFileFilter) can
be faster...
Posted by "Sebb (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NET-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebb resolved NET-455.
----------------------
Resolution: Not A Problem
Thanks for confirming that it is not a problem in NET
> FTPClient listFiles(FTPFileFilter) can be faster...
> ---------------------------------------------------
>
> Key: NET-455
> URL: https://issues.apache.org/jira/browse/NET-455
> Project: Commons Net
> Issue Type: Improvement
> Components: FTP
> Affects Versions: 3.0.1
> Environment: all
> Reporter: Nicolas
> Priority: Minor
> Labels: ftpclient, listFiles
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
> The first one is the conventionnal =>
> {code}
> files = ftpClient.listFiles("./", ftpFileFilter);
> {code}
> It takes about 90 seconds ...
> the second one is ... nearly as simple as the first one :
> {code}
> FTPFile [] allFilesListed = ftpClient.listFiles();
> files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
> {code}
> The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
> {code}
> public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
> ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
> for (FTPFile ftpf : ftpFileArray) {
> boolean accepted = filter.accept(ftpf);
> if (accepted) {
> listOfFTPFiles.add(ftpf);
> }
> }
> FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
> return listOfFTPFiles.toArray(resultArray);
> }
> {code}
> The second method is EIGHT times faster than the first one ... and I don't really understand why...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NET-455) FTPClient listFiles(FTPFileFilter) can
be faster...
Posted by "Sebb (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NET-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebb updated NET-455:
---------------------
Description:
I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
The first one is the conventionnal =>
{code}
files = ftpClient.listFiles("./", ftpFileFilter);
{code}
It takes about 90 seconds ...
the second one is ... nearly as simple as the first one :
{code}
FTPFile [] allFilesListed = ftpClient.listFiles();
files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
{code}
The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
{code}
public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
for (FTPFile ftpf : ftpFileArray) {
boolean accepted = filter.accept(ftpf);
if (accepted) {
listOfFTPFiles.add(ftpf);
}
}
FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
return listOfFTPFiles.toArray(resultArray);
}
{code}
The second method is EIGHT times faster than the first one ... and I don't really understand why...
was:
I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
The first one is the conventionnal => files = ftpClient.listFiles("./", ftpFileFilter);
It takes about 90 seconds ...
the second one is ... nearly as simple as the first one :
FTPFile [] allFilesListed = ftpClient.listFiles();
files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
for (FTPFile ftpf : ftpFileArray) {
boolean accepted = filter.accept(ftpf);
if (accepted) {
listOfFTPFiles.add(ftpf);
}
}
FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
return listOfFTPFiles.toArray(resultArray);
}
The second method is EIGHT times faster than the first one ... and I don't really understand why...
> FTPClient listFiles(FTPFileFilter) can be faster...
> ---------------------------------------------------
>
> Key: NET-455
> URL: https://issues.apache.org/jira/browse/NET-455
> Project: Commons Net
> Issue Type: Improvement
> Components: FTP
> Affects Versions: 3.0.1
> Environment: all
> Reporter: Nicolas
> Priority: Minor
> Labels: ftpclient, listFiles
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
> The first one is the conventionnal =>
> {code}
> files = ftpClient.listFiles("./", ftpFileFilter);
> {code}
> It takes about 90 seconds ...
> the second one is ... nearly as simple as the first one :
> {code}
> FTPFile [] allFilesListed = ftpClient.listFiles();
> files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
> {code}
> The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
> {code}
> public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
> ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
> for (FTPFile ftpf : ftpFileArray) {
> boolean accepted = filter.accept(ftpf);
> if (accepted) {
> listOfFTPFiles.add(ftpf);
> }
> }
> FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
> return listOfFTPFiles.toArray(resultArray);
> }
> {code}
> The second method is EIGHT times faster than the first one ... and I don't really understand why...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NET-455) FTPClient listFiles(FTPFileFilter) can
be faster...
Posted by "Nicolas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NET-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237019#comment-13237019 ]
Nicolas commented on NET-455:
-----------------------------
Thanks for your help !
The two cases where observed separately, and not in a row, so no caching possible. I did many runs like ABAABBA (but each time I stopped the JVM and restart my soft with the other filter method) and so I could reproduce the case each time.
However, I looked the source code of FTPClient, and you are right, no difference.
Maybe I have some sort of answer, but I'm really not sure :
I call
{code} files = ftpClient.listFiles("./", ftpFileFilter); {code}
So it calls the function
{code}initiateListParsing(String parserKey, String pathname) {code}
with the pathname not null, whereas in the "fast" case, the pathname is null.
So I do "LIST -a ./" instead of a "LIST -a" (if I'm right).
I'll dig into it on monday, trying null path or getFiles("./"), I'll come back with the results!
> FTPClient listFiles(FTPFileFilter) can be faster...
> ---------------------------------------------------
>
> Key: NET-455
> URL: https://issues.apache.org/jira/browse/NET-455
> Project: Commons Net
> Issue Type: Improvement
> Components: FTP
> Affects Versions: 3.0.1
> Environment: all
> Reporter: Nicolas
> Priority: Minor
> Labels: ftpclient, listFiles
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
> The first one is the conventionnal =>
> {code}
> files = ftpClient.listFiles("./", ftpFileFilter);
> {code}
> It takes about 90 seconds ...
> the second one is ... nearly as simple as the first one :
> {code}
> FTPFile [] allFilesListed = ftpClient.listFiles();
> files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
> {code}
> The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
> {code}
> public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
> ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
> for (FTPFile ftpf : ftpFileArray) {
> boolean accepted = filter.accept(ftpf);
> if (accepted) {
> listOfFTPFiles.add(ftpf);
> }
> }
> FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
> return listOfFTPFiles.toArray(resultArray);
> }
> {code}
> The second method is EIGHT times faster than the first one ... and I don't really understand why...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NET-455) FTPClient listFiles(FTPFileFilter) can
be faster...
Posted by "Nicolas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NET-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239370#comment-13239370 ]
Nicolas commented on NET-455:
-----------------------------
I did few tests !
I confirm that
{code} files = ftpClient.listFiles("./", ftpFileFilter); {code}
is slower than
{code} files = ftpClient.listFiles(null, ftpFileFilter); {code}
In mys tests, the first case took 3min05sec and the second one took 17 sec
However, this happens on a wuFTP Server 2.6.1 and can't be reproduced on a vsFTP Server. Maybe it's wuFTP related...
I tested
{code} files = ftpClient.listFiles(".", ftpFileFilter); {code} and it's as fast the null parameter (17sec). I don't know why listFiles "./" is slow on wuFTP. And of course, the manual filter with the ArrayList took 17.2/17.3 seconds (make sense after looking code).
I think this issue can be closed, however people have to remember to not use "./" with wuftp !
> FTPClient listFiles(FTPFileFilter) can be faster...
> ---------------------------------------------------
>
> Key: NET-455
> URL: https://issues.apache.org/jira/browse/NET-455
> Project: Commons Net
> Issue Type: Improvement
> Components: FTP
> Affects Versions: 3.0.1
> Environment: all
> Reporter: Nicolas
> Priority: Minor
> Labels: ftpclient, listFiles
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> I tested two ways of sorting files on a remote FTP server with many files ( > 110 000). Both of these use the FtpFileFilter class (with a test on the extension filename... very simple).
> The first one is the conventionnal =>
> {code}
> files = ftpClient.listFiles("./", ftpFileFilter);
> {code}
> It takes about 90 seconds ...
> the second one is ... nearly as simple as the first one :
> {code}
> FTPFile [] allFilesListed = ftpClient.listFiles();
> files = FTPFileFilterWithExtension.fastFilter(ftpFileFilter, allFilesListed);
> {code}
> The function FTPFileFilterWithExtension.fastFilter() call the same ftpFileFilter :
> {code}
> public static FTPFile [] fastFilter(FTPFileFilter filter, FTPFile ftpFileArray []) {
> ArrayList<FTPFile> listOfFTPFiles = new ArrayList<FTPFile>();
> for (FTPFile ftpf : ftpFileArray) {
> boolean accepted = filter.accept(ftpf);
> if (accepted) {
> listOfFTPFiles.add(ftpf);
> }
> }
> FTPFile resultArray [] = new FTPFile[listOfFTPFiles.size()];
> return listOfFTPFiles.toArray(resultArray);
> }
> {code}
> The second method is EIGHT times faster than the first one ... and I don't really understand why...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira