You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Leif Mortenson <le...@tanukisoftware.com> on 2004/08/17 17:41:52 UTC

[NET] [PATCH] FTPClient to work with FTP servers using a Japanese locale.

I ran into some problems attempting to get the FTPClient to be able to list
files on a Solaris server which is configured to use the Japanese locale.
The problem turned out to be caused by the regular expressions used to
parse the individual lines received from the server.

It works great with the English locale, but on Japanese servers, every
single
line of output was failing to match.

I modified the UnixFTPEntryParser to work correctly with either English
or Japanese locales. I also added appropriate test cases to make sure
every thing works.

Obviously other languages are going to most likely have similar problems
so there may be a slightly better regex that can be used to match them
as well. Just getting this to work took me long enough <:-)

On my machine, the Japanese text returned from the server was decoded
correctly. But I did not see exactly how it was deciding how to decode
the text. If I tried to access a Japanese server using an English client,
for example, I am not sure that things would work correctly. Sorry if
I overlooked it, but might there be a need for a way to set this encoding?

Patch follows

Thanks in advance,
Leif

cvs diff -u
? net/junit10154433.properties
cvs server: Diffing .
cvs server: Diffing net
cvs server: Diffing net/proposal
cvs server: Diffing net/proposal/ftp2
cvs server: Diffing net/proposal/ftp2/src
cvs server: Diffing net/proposal/ftp2/src/java
cvs server: Diffing net/proposal/ftp2/src/java/org
cvs server: Diffing net/proposal/ftp2/src/java/org/apache
cvs server: Diffing net/proposal/ftp2/src/java/org/apache/commons
cvs server: Diffing net/proposal/ftp2/src/java/org/apache/commons/net
cvs server: Diffing net/proposal/ftp2/src/java/org/apache/commons/net/ftp
cvs server: Diffing
net/proposal/ftp2/src/java/org/apache/commons/net/ftp/ftp2
cvs server: Diffing
net/proposal/ftp2/src/java/org/apache/commons/net/ftp/ftp2/parser
cvs server: Diffing net/proposal/ftp2/src/test
cvs server: Diffing net/proposal/ftp2/src/test/org
cvs server: Diffing net/proposal/ftp2/src/test/org/apache
cvs server: Diffing net/proposal/ftp2/src/test/org/apache/commons
cvs server: Diffing net/proposal/ftp2/src/test/org/apache/commons/net
cvs server: Diffing net/proposal/ftp2/src/test/org/apache/commons/net/ftp
cvs server: Diffing
net/proposal/ftp2/src/test/org/apache/commons/net/ftp/ftp2
cvs server: Diffing
net/proposal/ftp2/src/test/org/apache/commons/net/ftp/ftp2/parser
cvs server: Diffing net/src
cvs server: Diffing net/src/java
cvs server: Diffing net/src/java/examples
cvs server: Diffing net/src/java/org
cvs server: Diffing net/src/java/org/apache
cvs server: Diffing net/src/java/org/apache/commons
cvs server: Diffing net/src/java/org/apache/commons/net
cvs server: Diffing net/src/java/org/apache/commons/net/bsd
cvs server: Diffing net/src/java/org/apache/commons/net/ftp
cvs server: Diffing net/src/java/org/apache/commons/net/ftp/parser
Index:
net/src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java,v
retrieving revision 1.18
diff -u -r1.18 UnixFTPEntryParser.java
---
net/src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java
28 Jul 2004 05:01:47 -0000 1.18
+++
net/src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java
17 Aug 2004 15:26:58 -0000
@@ -35,7 +35,7 @@
* to determine which month is matched by the parser
*/
private static final String MONTHS =
- "(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)";
+ "Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec";

/**
* this is the regular expression used by this parser.
@@ -63,9 +63,9 @@
+ "(\\S+)\\s+"
+ "(?:(\\S+)\\s+)?"
+ "(\\d+)\\s+"
- + MONTHS + "\\s+"
- + "((?:[0-9])|(?:[0-2][0-9])|(?:3[0-1]))\\s+"
- + "((\\d\\d\\d\\d)|((?:[01]\\d)|(?:2[0123])|(?:[1-9])):([012345]\\d))\\s+"
+ + "((?:" + MONTHS + ")|(?:(?:(?:[0-9])|(?:[0-1][0-9]))))\u6708?\\s+"
+ + "(?:((?:[0-9])|(?:[0-2][0-9])|(?:3[0-1]))(?:\u65e5?))\\s+"
+ +
"((?:(\\d\\d\\d\\d)\u5e74?)|((?:[01]\\d)|(?:2[0123])|(?:[1-9])):([012345]\\d))\\s+"
+ "(\\S+)(\\s*.*)";


@@ -94,7 +94,6 @@
*/
public FTPFile parseFTPEntry(String entry)
{
-
FTPFile file = new FTPFile();
file.setRawListing(entry);
int type;
@@ -130,8 +129,8 @@
// break; - fall through
case 'f':
case '-':
- type = FTPFile.FILE_TYPE;
- break;
+ type = FTPFile.FILE_TYPE;
+ break;
default:
type = FTPFile.UNKNOWN_TYPE;
}
@@ -189,8 +188,17 @@

try
{
+ int month;
int pos = MONTHS.indexOf(mo);
- int month = pos / 4;
+ if ( pos >= 0 )
+ {
+ month = pos / 4;
+ }
+ else
+ {
+ // Expected to be base-0
+ month = Integer.parseInt( mo ) - 1;
+ }

if (null != yr)
{
cvs server: Diffing net/src/java/org/apache/commons/net/io
cvs server: Diffing net/src/java/org/apache/commons/net/nntp
cvs server: Diffing net/src/java/org/apache/commons/net/pop3
cvs server: Diffing net/src/java/org/apache/commons/net/smtp
cvs server: Diffing net/src/java/org/apache/commons/net/telnet
cvs server: Diffing net/src/java/org/apache/commons/net/tftp
cvs server: Diffing net/src/java/org/apache/commons/net/util
cvs server: Diffing net/src/test
cvs server: Diffing net/src/test/org
cvs server: Diffing net/src/test/org/apache
cvs server: Diffing net/src/test/org/apache/commons
cvs server: Diffing net/src/test/org/apache/commons/net
cvs server: Diffing net/src/test/org/apache/commons/net/ftp
cvs server: Diffing net/src/test/org/apache/commons/net/ftp/parser
Index:
net/src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java,v
retrieving revision 1.14
diff -u -r1.14 UnixFTPEntryParserTest.java
---
net/src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java
29 Jul 2004 11:38:36 -0000 1.14
+++
net/src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java
17 Aug 2004 15:26:58 -0000
@@ -14,7 +14,6 @@
* limitations under the License.
*/
package org.apache.commons.net.ftp.parser;
-
import java.util.Calendar;

import junit.framework.TestSuite;
@@ -69,7 +68,9 @@
"-rwxr-xr-t 1 500 500 0 Mar 25 08:21 testStickyExec",
"-rwSr-Sr-- 1 500 500 0 Mar 25 08:22 testSuid",
"-rwsr-sr-- 1 500 500 0 Mar 25 08:23 testSuidExec",
- "-rw-r--r-- 1 1 3518644 May 25 12:12 std"
+ "-rw-r--r-- 1 1 3518644 May 25 12:12 std",
+ "-rw-rw---- 1 ep1adm sapsys 0 8\u6708 17\u65e5 20:10 caerrinf",
+ "-rw-rw---- 1 ep1adm sapsys 0 6\u6708 3\u65e5 2003\u5e74
\u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv"
};

/**
@@ -215,6 +216,78 @@
cal.set(Calendar.DATE,2);
cal.set(Calendar.HOUR_OF_DAY, 15);
cal.set(Calendar.MINUTE, 13);
+ assertEquals(df.format(cal.getTime()),
+ df.format(f.getTimestamp().getTime()));
+ }
+
+ /**
+ * @see
org.apache.commons.net.ftp.parser.FTPParseTestFramework#testParseFieldsOnFile()
+ */
+ public void testParseFieldsOnFileJapaneseTime() throws Exception
+ {
+ FTPFile f = getParser().parseFTPEntry("-rwxr-xr-x 2 user group 4096
3\u6708 2\u65e5 15:13 zxbox");
+ assertNotNull("Could not parse entry.",
+ f);
+ assertTrue("Should have been a file.",
+ f.isFile());
+ checkPermissions(f);
+ assertEquals(2,
+ f.getHardLinkCount());
+ assertEquals("user",
+ f.getUser());
+ assertEquals("group",
+ f.getGroup());
+ assertEquals("zxbox",
+ f.getName());
+ assertEquals(4096,
+ f.getSize());
+
+ Calendar cal = Calendar.getInstance();
+ cal.set(Calendar.MONTH, Calendar.MARCH);
+
+ cal.set(Calendar.DATE,1);
+ cal.set(Calendar.HOUR_OF_DAY, 0);
+ cal.set(Calendar.MINUTE, 0);
+ cal.set(Calendar.SECOND, 0);
+ if (f.getTimestamp().getTime().before(cal.getTime())) {
+ cal.add(Calendar.YEAR, -1);
+ }
+ cal.set(Calendar.DATE,2);
+ cal.set(Calendar.HOUR_OF_DAY, 15);
+ cal.set(Calendar.MINUTE, 13);
+ assertEquals(df.format(cal.getTime()),
+ df.format(f.getTimestamp().getTime()));
+ }
+
+ /**
+ * @see
org.apache.commons.net.ftp.parser.FTPParseTestFramework#testParseFieldsOnFile()
+ */
+ public void testParseFieldsOnFileJapaneseYear() throws Exception
+ {
+ FTPFile f = getParser().parseFTPEntry("-rwxr-xr-x 2 user group 4096
3\u6708 2\u65e5 2003\u5e74 \u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv");
+ assertNotNull("Could not parse entry.",
+ f);
+ assertTrue("Should have been a file.",
+ f.isFile());
+ checkPermissions(f);
+ assertEquals(2,
+ f.getHardLinkCount());
+ assertEquals("user",
+ f.getUser());
+ assertEquals("group",
+ f.getGroup());
+ assertEquals("\u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv",
+ f.getName());
+ assertEquals(4096,
+ f.getSize());
+
+ Calendar cal = Calendar.getInstance();
+ cal.set(Calendar.YEAR, 2003);
+ cal.set(Calendar.MONTH, Calendar.MARCH);
+ cal.set(Calendar.DATE,2);
+ cal.set(Calendar.HOUR_OF_DAY, 0);
+ cal.set(Calendar.MINUTE, 0);
+ cal.set(Calendar.SECOND, 0);
assertEquals(df.format(cal.getTime()),
df.format(f.getTimestamp().getTime()));
}
cvs server: Diffing net/src/test/org/apache/commons/net/telnet
cvs server: Diffing net/xdocs
cvs server: Diffing net/xdocs/images



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org

Re: [NET] [PATCH] FTPClient to work with FTP servers using a Japanese locale. [UPDATE]

Posted by Steve Cohen <sc...@javactivity.org>.
A diff -u patch file is best.  Please don't send the actual files.
cvs diff -u [files] >> patchfile.txt

On Wednesday 18 August 2004 9:02 pm, Leif Mortenson wrote:
> Steve,
> No problem. Let me know if you would prefer a different format or even the
> files themselves. They are updated to the latest CVS head.
>
> Cheers,
> Leif
>
> Steve Cohen wrote:
> >On Wednesday 18 August 2004 1:14 am, Leif Mortenson wrote:
> >
> >Leif, could I possibly trouble you to submit your patch as an attached
> > file rather than inline in the email?  I've grown a bit lazy and Eclipse
> > doesn't like the snippets I try to patch in via the clipboard.
> >
> >Steve


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org

Re: [NET] [PATCH] FTPClient to work with FTP servers using a Japanese locale. [UPDATE]

Posted by Leif Mortenson <le...@tanukisoftware.com>.
Steve,
No problem. Let me know if you would prefer a different format or even the
files themselves. They are updated to the latest CVS head.

Cheers,
Leif

Steve Cohen wrote:

>On Wednesday 18 August 2004 1:14 am, Leif Mortenson wrote:
>
>Leif, could I possibly trouble you to submit your patch as an attached file 
>rather than inline in the email?  I've grown a bit lazy and Eclipse doesn't 
>like the snippets I try to patch in via the clipboard.
>
>Steve
>  
>

Re: [NET] [PATCH] FTPClient to work with FTP servers using a Japanese locale. [UPDATE]

Posted by Steve Cohen <sc...@javactivity.org>.
On Wednesday 18 August 2004 1:14 am, Leif Mortenson wrote:

Leif, could I possibly trouble you to submit your patch as an attached file 
rather than inline in the email?  I've grown a bit lazy and Eclipse doesn't 
like the snippets I try to patch in via the clipboard.

Steve

> Doing some testing with other Japanese FTP servers, I found that I was
> correct
> about there being lurking problems with character encodings. The FTPClient
> does provide a method to set the control encoding: setControlEncoding().
> But that encoding was not being used in a lot of places.
>
> I had to add a couple new methods to handle encoding, but all existing
> public
> methods were left unchanged and function the same as they did before.
>
> I went through and made several more changes to make use of the configured
> control encoding.
>
> In addition, I also modified the regex that I posted yesterday so that
> it should
> now work with languages other Japanese. I have not tested other languages,
> but any that place a month, day, or year unit after a numerical data
> element should now work correctly. Other languages that use non-english
> month abbreviations will still not work.
>
> In addition, if the user specified an invalid control encoding when used
> with the
> Ant task, it was kicking out some some nasty NPEs because the ant task
> always
> tries to log off. That could be viewed as a problem with the Ant task,
> but I made
> the FTP class handle this case more gracefully.
> The problem is that the isConnected method returns true when the socket has
> been opened, but the protocol is not yet all the way initialized. The
> logout and disconnect methods now handle these cases correctly.
>
> The new patch which also includes yesterday's changes follows at the end of
> this mail.
>
> Cheers,
> Leif
>



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org

[NET] [PATCH] FTPClient to work with FTP servers using a Japanese locale. [UPDATE]

Posted by Leif Mortenson <le...@tanukisoftware.com>.
Doing some testing with other Japanese FTP servers, I found that I was
correct
about there being lurking problems with character encodings. The FTPClient
does provide a method to set the control encoding: setControlEncoding().
But that encoding was not being used in a lot of places.

I had to add a couple new methods to handle encoding, but all existing
public
methods were left unchanged and function the same as they did before.

I went through and made several more changes to make use of the configured
control encoding.

In addition, I also modified the regex that I posted yesterday so that
it should
now work with languages other Japanese. I have not tested other languages,
but any that place a month, day, or year unit after a numerical data element
should now work correctly. Other languages that use non-english month
abbreviations will still not work.

In addition, if the user specified an invalid control encoding when used
with the
Ant task, it was kicking out some some nasty NPEs because the ant task
always
tries to log off. That could be viewed as a problem with the Ant task,
but I made
the FTP class handle this case more gracefully.
The problem is that the isConnected method returns true when the socket has
been opened, but the protocol is not yet all the way initialized. The logout
and disconnect methods now handle these cases correctly.

The new patch which also includes yesterday's changes follows at the end of
this mail.

Cheers,
Leif


Leif Mortenson wrote:

>I ran into some problems attempting to get the FTPClient to be able to list
>files on a Solaris server which is configured to use the Japanese locale.
>The problem turned out to be caused by the regular expressions used to
>parse the individual lines received from the server.
>
>It works great with the English locale, but on Japanese servers, every
>single
>line of output was failing to match.
>
>I modified the UnixFTPEntryParser to work correctly with either English
>or Japanese locales. I also added appropriate test cases to make sure
>every thing works.
>
>Obviously other languages are going to most likely have similar problems
>so there may be a slightly better regex that can be used to match them
>as well. Just getting this to work took me long enough <:-)
>
>On my machine, the Japanese text returned from the server was decoded
>correctly. But I did not see exactly how it was deciding how to decode
>the text. If I tried to access a Japanese server using an English client,
>for example, I am not sure that things would work correctly. Sorry if
>I overlooked it, but might there be a need for a way to set this encoding?
>
>Patch follows
>
>Thanks in advance,
>Leif
>  
>
<old patch snipped>

New path:

cvs diff -u
cvs server: Diffing .
cvs server: Diffing proposal
cvs server: Diffing proposal/ftp2
cvs server: Diffing proposal/ftp2/src
cvs server: Diffing proposal/ftp2/src/java
cvs server: Diffing proposal/ftp2/src/java/org
cvs server: Diffing proposal/ftp2/src/java/org/apache
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons/net
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons/net/ftp
cvs server: Diffing proposal/ftp2/src/java/org/apache/commons/net/ftp/ftp2
cvs server: Diffing
proposal/ftp2/src/java/org/apache/commons/net/ftp/ftp2/parser
cvs server: Diffing proposal/ftp2/src/test
cvs server: Diffing proposal/ftp2/src/test/org
cvs server: Diffing proposal/ftp2/src/test/org/apache
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons/net
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons/net/ftp
cvs server: Diffing proposal/ftp2/src/test/org/apache/commons/net/ftp/ftp2
cvs server: Diffing
proposal/ftp2/src/test/org/apache/commons/net/ftp/ftp2/parser
cvs server: Diffing src
cvs server: Diffing src/java
cvs server: Diffing src/java/examples
cvs server: Diffing src/java/org
cvs server: Diffing src/java/org/apache
cvs server: Diffing src/java/org/apache/commons
cvs server: Diffing src/java/org/apache/commons/net
cvs server: Diffing src/java/org/apache/commons/net/bsd
cvs server: Diffing src/java/org/apache/commons/net/ftp
Index: src/java/org/apache/commons/net/ftp/FTP.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTP.java,v
retrieving revision 1.16
diff -u -r1.16 FTP.java
--- src/java/org/apache/commons/net/ftp/FTP.java 8 Aug 2004 20:38:35
-0000 1.16
+++ src/java/org/apache/commons/net/ftp/FTP.java 18 Aug 2004 06:11:31 -0000
@@ -420,6 +420,11 @@
***/
public int sendCommand(String command, String args) throws IOException
{
+ if ((__commandBuffer == null) || (_controlOutput == null))
+ {
+ throw new IOException("Not connected.");
+ }
+
String message;

__commandBuffer.setLength(0);
Index: src/java/org/apache/commons/net/ftp/FTPClient.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPClient.java,v
retrieving revision 1.38
diff -u -r1.38 FTPClient.java
--- src/java/org/apache/commons/net/ftp/FTPClient.java 29 Jun 2004
04:54:30 -0000 1.38
+++ src/java/org/apache/commons/net/ftp/FTPClient.java 18 Aug 2004
06:11:32 -0000
@@ -1931,8 +1931,8 @@
if ((socket = _openDataConnection_(FTPCommand.NLST, pathname)) == null)
return null;

- reader =
- new BufferedReader(new InputStreamReader(socket.getInputStream()));
+ reader = new BufferedReader(
+ new InputStreamReader(socket.getInputStream(), getControlEncoding()));

results = new Vector();
while ((line = reader.readLine()) != null)
@@ -2338,7 +2338,7 @@
}


- engine.readServerList(socket.getInputStream());
+ engine.readServerList(socket.getInputStream(), getControlEncoding());

socket.close();

@@ -2425,7 +2425,7 @@
if ((socket = _openDataConnection_(FTPCommand.LIST, pathname)) == null)
return new FTPFile[0];

- results = parser.parseFileList(socket.getInputStream());
+ results = parser.parseFileList(socket.getInputStream(),
getControlEncoding());

socket.close();

Index: src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java,v
retrieving revision 1.8
diff -u -r1.8 FTPFileEntryParserImpl.java
--- src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java 22
Apr 2004 00:48:07 -0000 1.8
+++ src/java/org/apache/commons/net/ftp/FTPFileEntryParserImpl.java 18
Aug 2004 06:11:32 -0000
@@ -34,7 +34,7 @@
* The constructor for a FTPFileEntryParserImpl object.
*/
public FTPFileEntryParserImpl()
- {
+ {
}


@@ -47,16 +47,37 @@
* <p>
* @param listStream The InputStream from which the file list should be
* read.
+ * @param encoding The encoding to use, null for system default.
* @return The list of file information contained in the given path. null
* if the list could not be obtained or if there are no files in
* the directory.
* @exception java.io.IOException If an I/O error occurs reading the
listStream.
***/
- public FTPFile[] parseFileList(InputStream listStream) throws IOException
+ public FTPFile[] parseFileList(InputStream listStream, String
encoding) throws IOException
{
- FTPFileList ffl = FTPFileList.create(listStream, this);
+ FTPFileList ffl = FTPFileList.create(listStream, this, encoding);
return ffl.getFiles();
+ }

+ /***
+ * Parses an FTP server file listing and converts it into a usable format
+ * in the form of an array of <code> FTPFile </code> instances. If the
+ * file list contains no files, <code> null </code> should be
+ * returned, otherwise an array of <code> FTPFile </code> instances
+ * representing the files in the directory is returned.
+ * <p>
+ * @param listStream The InputStream from which the file list should be
+ * read.
+ * @return The list of file information contained in the given path. null
+ * if the list could not be obtained or if there are no files in
+ * the directory.
+ * @exception java.io.IOException If an I/O error occurs reading the
listStream.
+ *
+ * @deprecated The version of this method which takes an encoding
should be used.
+ ***/
+ public FTPFile[] parseFileList(InputStream listStream) throws IOException
+ {
+ return parseFileList(listStream, null);
}

/**
Index: src/java/org/apache/commons/net/ftp/FTPFileList.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPFileList.java,v
retrieving revision 1.13
diff -u -r1.13 FTPFileList.java
--- src/java/org/apache/commons/net/ftp/FTPFileList.java 21 Apr 2004
23:30:33 -0000 1.13
+++ src/java/org/apache/commons/net/ftp/FTPFileList.java 18 Aug 2004
06:11:32 -0000
@@ -82,6 +82,7 @@
* the output of the LIST command was returned
* @param parser the default <code>FTPFileEntryParser</code> to be used
* by this object. This may later be changed using the init() method.
+ * @param encoding The encoding to use, null for system default.
*
* @return the <code>FTPFileList</code> created, with an initialized
* of unparsed lines of output. Will be null if the listing cannot
@@ -90,26 +91,61 @@
* Thrown on any failure to read from the socket.
*/
public static FTPFileList create(InputStream stream,
- FTPFileEntryParser parser)
+ FTPFileEntryParser parser,
+ String encoding)
throws IOException
{
FTPFileList list = new FTPFileList(parser);
- list.readStream(stream);
+ list.readStream(stream, encoding);
parser.preParse(list.lines);
return list;
}

/**
+ * The only way to create an <code>FTPFileList</code> object. Invokes
+ * the private constructor and then reads the stream supplied stream to
+ * build the intermediate array of "lines" which will later be parsed
+ * into <code>FTPFile</code> object.
+ *
+ * @param stream The input stream created by reading the socket on which
+ * the output of the LIST command was returned
+ * @param parser the default <code>FTPFileEntryParser</code> to be used
+ * by this object. This may later be changed using the init() method.
+ *
+ * @return the <code>FTPFileList</code> created, with an initialized
+ * of unparsed lines of output. Will be null if the listing cannot
+ * be read from the stream.
+ * @exception IOException
+ * Thrown on any failure to read from the socket.
+ *
+ * @deprecated The version of this method which takes an encoding
should be used.
+ */
+ public static FTPFileList create(InputStream stream,
+ FTPFileEntryParser parser)
+ throws IOException
+ {
+ return create(stream, parser, null);
+ }
+
+ /**
* internal method for reading the input into the <code>lines</code> vector.
*
* @param stream The socket stream on which the input will be read.
+ * @param encoding The encoding to use, null for system default.
*
* @exception IOException thrown on any failure to read the stream
*/
- public void readStream(InputStream stream) throws IOException
+ public void readStream(InputStream stream, String encoding) throws
IOException
{
- BufferedReader reader =
- new BufferedReader(new InputStreamReader(stream));
+ BufferedReader reader;
+ if (encoding == null)
+ {
+ reader = new BufferedReader(new InputStreamReader(stream));
+ }
+ else
+ {
+ reader = new BufferedReader(new InputStreamReader(stream, encoding));
+ }

String line = this.parser.readNextEntry(reader);

@@ -119,6 +155,20 @@
line = this.parser.readNextEntry(reader);
}
reader.close();
+ }
+
+ /**
+ * internal method for reading the input into the <code>lines</code>
vector.
+ *
+ * @param stream The socket stream on which the input will be read.
+ *
+ * @exception IOException thrown on any failure to read the stream
+ *
+ * @deprecated The version of this method which takes an encoding
should be used.
+ */
+ public void readStream(InputStream stream) throws IOException
+ {
+ readStream(stream, null);
}

/**
Index: src/java/org/apache/commons/net/ftp/FTPFileListParser.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPFileListParser.java,v
retrieving revision 1.12
diff -u -r1.12 FTPFileListParser.java
--- src/java/org/apache/commons/net/ftp/FTPFileListParser.java 29 Jun
2004 02:16:33 -0000 1.12
+++ src/java/org/apache/commons/net/ftp/FTPFileListParser.java 18 Aug
2004 06:11:32 -0000
@@ -45,10 +45,29 @@
* <p>
* @param listStream The InputStream from which the file list should be
* read.
+ * @param encoding The encoding to use, null for system default.
* @return The list of file information contained in the given path. null
* if the list could not be obtained or if there are no files in
* the directory.
* @exception IOException If an I/O error occurs reading the listStream.
+ ***/
+ FTPFile[] parseFileList(InputStream listStream, String encoding)
throws IOException;
+
+ /***
+ * Parses an FTP server file listing and converts it into a usable format
+ * in the form of an array of <code> FTPFile </code> instances. If the
+ * file list contains no files, <code> null </code> should be
+ * returned, otherwise an array of <code> FTPFile </code> instances
+ * representing the files in the directory is returned.
+ * <p>
+ * @param listStream The InputStream from which the file list should be
+ * read.
+ * @return The list of file information contained in the given path. null
+ * if the list could not be obtained or if there are no files in
+ * the directory.
+ * @exception IOException If an I/O error occurs reading the listStream.
+ *
+ * @deprecated The version of this method which takes an encoding
should be used.
***/
FTPFile[] parseFileList(InputStream listStream) throws IOException;

Index: src/java/org/apache/commons/net/ftp/FTPListParseEngine.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/FTPListParseEngine.java,v
retrieving revision 1.7
diff -u -r1.7 FTPListParseEngine.java
--- src/java/org/apache/commons/net/ftp/FTPListParseEngine.java 22 Apr
2004 00:48:07 -0000 1.7
+++ src/java/org/apache/commons/net/ftp/FTPListParseEngine.java 18 Aug
2004 06:11:33 -0000
@@ -87,19 +87,39 @@
* on the server.
*
* @param stream input stream provided by the server socket.
+ * @param encoding The encoding to use, null for system default.
*
* @exception IOException
* thrown on any failure to read from the sever.
*/
- public void readServerList(InputStream stream)
+ public void readServerList(InputStream stream, String encoding)
throws IOException
{
this.entries = new LinkedList();
- readStream(stream);
+ readStream(stream, encoding);
this.parser.preParse(this.entries);
resetIterator();
}

+ /**
+ * handle the iniitial reading and preparsing of the list returned by
+ * the server. After this method has completed, this object will contain
+ * a list of unparsed entries (Strings) each referring to a unique file
+ * on the server.
+ *
+ * @param stream input stream provided by the server socket.
+ *
+ * @exception IOException
+ * thrown on any failure to read from the sever.
+ *
+ * @deprecated The version of this method which takes an encoding
should be used.
+ */
+ public void readServerList(InputStream stream)
+ throws IOException
+ {
+ readServerList(stream, null);
+ }
+

/**
* Internal method for reading the input into the <code>entries</code> list.
@@ -110,14 +130,22 @@
* and other data that will not be part of the final listing.
*
* @param stream The socket stream on which the input will be read.
+ * @param encoding The encoding to use, null for system default.
*
* @exception IOException
* thrown on any failure to read the stream
*/
- private void readStream(InputStream stream) throws IOException
+ private void readStream(InputStream stream, String encoding) throws
IOException
{
- BufferedReader reader =
- new BufferedReader(new InputStreamReader(stream));
+ BufferedReader reader;
+ if (encoding == null)
+ {
+ reader = new BufferedReader(new InputStreamReader(stream));
+ }
+ else
+ {
+ reader = new BufferedReader(new InputStreamReader(stream, encoding));
+ }

String line = this.parser.readNextEntry(reader);

cvs server: Diffing src/java/org/apache/commons/net/ftp/parser
Index: src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java,v
retrieving revision 1.18
diff -u -r1.18 UnixFTPEntryParser.java
--- src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java
28 Jul 2004 05:01:47 -0000 1.18
+++ src/java/org/apache/commons/net/ftp/parser/UnixFTPEntryParser.java
18 Aug 2004 06:11:33 -0000
@@ -35,7 +35,7 @@
* to determine which month is matched by the parser
*/
private static final String MONTHS =
- "(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)";
+ "Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec";

/**
* this is the regular expression used by this parser.
@@ -55,6 +55,15 @@
* execution is on
* T the 1000 bit is turned on, and execution is off (undefined bit-
* state)
+ *
+ * Japanese formatted listings use numerical months, and the month, day
+ * and year are each followed by a single kanji character. Rather than
+ * testing for the specific kanji characters, this expression allows any
+ * non-numerical, non-space character(s). This should make other languages
+ * which use a similar syntax work as well.
+ *
+ * (For Japanese, the matched characters are '\u6708' after month,
+ * '\u65e5' after day, and '\u5e74' after year.)
*/
private static final String REGEX =
"([bcdlfmpSs-])"
@@ -63,9 +72,9 @@
+ "(\\S+)\\s+"
+ "(?:(\\S+)\\s+)?"
+ "(\\d+)\\s+"
- + MONTHS + "\\s+"
- + "((?:[0-9])|(?:[0-2][0-9])|(?:3[0-1]))\\s+"
- + "((\\d\\d\\d\\d)|((?:[01]\\d)|(?:2[0123])|(?:[1-9])):([012345]\\d))\\s+"
+ + "((?:" + MONTHS + ")|(?:(?:(?:[0-9])|(?:[0-1][0-9]))))[^0-9\\s]*?\\s+"
+ + "(?:((?:[0-9])|(?:[0-2][0-9])|(?:3[0-1]))(?:[^0-9\\s]*?))\\s+"
+ +
"((?:(\\d\\d\\d\\d)[^0-9\\s]*?)|((?:[01]\\d)|(?:2[0123])|(?:[1-9])):([012345]\\d))\\s+"
+ "(\\S+)(\\s*.*)";


@@ -94,7 +103,6 @@
*/
public FTPFile parseFTPEntry(String entry)
{
-
FTPFile file = new FTPFile();
file.setRawListing(entry);
int type;
@@ -114,7 +122,7 @@
String min = group(24);
String name = group(25);
String endtoken = group(26);
-
+
// bcdlfmpSs-
switch (typeStr.charAt(0))
{
@@ -130,8 +138,8 @@
// break; - fall through
case 'f':
case '-':
- type = FTPFile.FILE_TYPE;
- break;
+ type = FTPFile.FILE_TYPE;
+ break;
default:
type = FTPFile.UNKNOWN_TYPE;
}
@@ -189,8 +197,17 @@

try
{
+ int month;
int pos = MONTHS.indexOf(mo);
- int month = pos / 4;
+ if ( pos >= 0 )
+ {
+ month = pos / 4;
+ }
+ else
+ {
+ // Expected to be base-0
+ month = Integer.parseInt( mo ) - 1;
+ }

if (null != yr)
{
Index: src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java,v
retrieving revision 1.24
diff -u -r1.24 VMSFTPEntryParser.java
--- src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java 28
Jul 2004 05:01:47 -0000 1.24
+++ src/java/org/apache/commons/net/ftp/parser/VMSFTPEntryParser.java 18
Aug 2004 06:11:33 -0000
@@ -94,18 +94,17 @@
* <p>
* @param listStream The InputStream from which the file list should be
* read.
+ * @param encoding The encoding to use, null for system default.
* @return The list of file information contained in the given path. null
* if the list could not be obtained or if there are no files in
* the directory.
* @exception IOException If an I/O error occurs reading the listStream.
***/
- public FTPFile[] parseFileList(InputStream listStream) throws
IOException {
+ public FTPFile[] parseFileList(InputStream listStream, String
encoding) throws IOException {
FTPListParseEngine engine = new FTPListParseEngine(this);
- engine.readServerList(listStream);
+ engine.readServerList(listStream, encoding);
return engine.getFiles();
}
-
-

/**
* Parses a line of a VMS FTP server file listing and converts it into a
cvs server: Diffing src/java/org/apache/commons/net/io
cvs server: Diffing src/java/org/apache/commons/net/nntp
cvs server: Diffing src/java/org/apache/commons/net/pop3
cvs server: Diffing src/java/org/apache/commons/net/smtp
cvs server: Diffing src/java/org/apache/commons/net/telnet
cvs server: Diffing src/java/org/apache/commons/net/tftp
cvs server: Diffing src/java/org/apache/commons/net/util
cvs server: Diffing src/test
cvs server: Diffing src/test/org
cvs server: Diffing src/test/org/apache
cvs server: Diffing src/test/org/apache/commons
cvs server: Diffing src/test/org/apache/commons/net
cvs server: Diffing src/test/org/apache/commons/net/ftp
cvs server: Diffing src/test/org/apache/commons/net/ftp/parser
Index:
src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java,v
retrieving revision 1.14
diff -u -r1.14 UnixFTPEntryParserTest.java
---
src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java
29 Jul 2004 11:38:36 -0000 1.14
+++
src/test/org/apache/commons/net/ftp/parser/UnixFTPEntryParserTest.java
18 Aug 2004 06:11:33 -0000
@@ -14,7 +14,6 @@
* limitations under the License.
*/
package org.apache.commons.net.ftp.parser;
-
import java.util.Calendar;

import junit.framework.TestSuite;
@@ -69,7 +68,9 @@
"-rwxr-xr-t 1 500 500 0 Mar 25 08:21 testStickyExec",
"-rwSr-Sr-- 1 500 500 0 Mar 25 08:22 testSuid",
"-rwsr-sr-- 1 500 500 0 Mar 25 08:23 testSuidExec",
- "-rw-r--r-- 1 1 3518644 May 25 12:12 std"
+ "-rw-r--r-- 1 1 3518644 May 25 12:12 std",
+ "-rw-rw---- 1 ep1adm sapsys 0 8\u6708 17\u65e5 20:10 caerrinf",
+ "-rw-rw---- 1 ep1adm sapsys 0 6\u6708 3\u65e5 2003\u5e74
\u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv"
};

/**
@@ -215,6 +216,78 @@
cal.set(Calendar.DATE,2);
cal.set(Calendar.HOUR_OF_DAY, 15);
cal.set(Calendar.MINUTE, 13);
+ assertEquals(df.format(cal.getTime()),
+ df.format(f.getTimestamp().getTime()));
+ }
+
+ /**
+ * @see
org.apache.commons.net.ftp.parser.FTPParseTestFramework#testParseFieldsOnFile()
+ */
+ public void testParseFieldsOnFileJapaneseTime() throws Exception
+ {
+ FTPFile f = getParser().parseFTPEntry("-rwxr-xr-x 2 user group 4096
3\u6708 2\u65e5 15:13 zxbox");
+ assertNotNull("Could not parse entry.",
+ f);
+ assertTrue("Should have been a file.",
+ f.isFile());
+ checkPermissions(f);
+ assertEquals(2,
+ f.getHardLinkCount());
+ assertEquals("user",
+ f.getUser());
+ assertEquals("group",
+ f.getGroup());
+ assertEquals("zxbox",
+ f.getName());
+ assertEquals(4096,
+ f.getSize());
+
+ Calendar cal = Calendar.getInstance();
+ cal.set(Calendar.MONTH, Calendar.MARCH);
+
+ cal.set(Calendar.DATE,1);
+ cal.set(Calendar.HOUR_OF_DAY, 0);
+ cal.set(Calendar.MINUTE, 0);
+ cal.set(Calendar.SECOND, 0);
+ if (f.getTimestamp().getTime().before(cal.getTime())) {
+ cal.add(Calendar.YEAR, -1);
+ }
+ cal.set(Calendar.DATE,2);
+ cal.set(Calendar.HOUR_OF_DAY, 15);
+ cal.set(Calendar.MINUTE, 13);
+ assertEquals(df.format(cal.getTime()),
+ df.format(f.getTimestamp().getTime()));
+ }
+
+ /**
+ * @see
org.apache.commons.net.ftp.parser.FTPParseTestFramework#testParseFieldsOnFile()
+ */
+ public void testParseFieldsOnFileJapaneseYear() throws Exception
+ {
+ FTPFile f = getParser().parseFTPEntry("-rwxr-xr-x 2 user group 4096
3\u6708 2\u65e5 2003\u5e74 \u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv");
+ assertNotNull("Could not parse entry.",
+ f);
+ assertTrue("Should have been a file.",
+ f.isFile());
+ checkPermissions(f);
+ assertEquals(2,
+ f.getHardLinkCount());
+ assertEquals("user",
+ f.getUser());
+ assertEquals("group",
+ f.getGroup());
+ assertEquals("\u8a66\u9a13\u30d5\u30a1\u30a4\u30eb.csv",
+ f.getName());
+ assertEquals(4096,
+ f.getSize());
+
+ Calendar cal = Calendar.getInstance();
+ cal.set(Calendar.YEAR, 2003);
+ cal.set(Calendar.MONTH, Calendar.MARCH);
+ cal.set(Calendar.DATE,2);
+ cal.set(Calendar.HOUR_OF_DAY, 0);
+ cal.set(Calendar.MINUTE, 0);
+ cal.set(Calendar.SECOND, 0);
assertEquals(df.format(cal.getTime()),
df.format(f.getTimestamp().getTime()));
}
Index: src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java
===================================================================
RCS file:
/home/cvspublic/jakarta-commons/net/src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java,v
retrieving revision 1.16
diff -u -r1.16 VMSFTPEntryParserTest.java
---
src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java 29
Jun 2004 04:54:32 -0000 1.16
+++
src/test/org/apache/commons/net/ftp/parser/VMSFTPEntryParserTest.java 18
Aug 2004 06:11:33 -0000
@@ -91,7 +91,7 @@
VMSFTPEntryParser parser = new VMSFTPEntryParser();
FTPListParseEngine engine = new FTPListParseEngine(parser);
engine.readServerList(
- new ByteArrayInputStream(fullListing.getBytes()));
+ new ByteArrayInputStream(fullListing.getBytes()), "ISO-8859-1");
FTPFile[] files = engine.getFiles();
assertEquals(6, files.length);
assertFileInListing(files, "2-JUN.LIS");
@@ -111,7 +111,7 @@
VMSFTPEntryParser parser = new VMSVersioningFTPEntryParser();
FTPListParseEngine engine = new FTPListParseEngine(parser);
engine.readServerList(
- new ByteArrayInputStream(fullListing.getBytes()));
+ new ByteArrayInputStream(fullListing.getBytes()), "ISO-8859-1");
FTPFile[] files = engine.getFiles();
assertEquals(3, files.length);
assertFileInListing(files, "1-JUN.LIS;1");
cvs server: Diffing src/test/org/apache/commons/net/telnet
cvs server: Diffing xdocs
cvs server: Diffing xdocs/images



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org