You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Oliver Zemann <ol...@gmail.com> on 2016/08/28 13:50:36 UTC
NPE in getRemoteAdress
Hi
i am trying to get some files from a FTP Server. When i use
retrieveFile(), it works, but i would like to add some listeners to it
(how many bytes got transfered etc.) so i used retrieveFileStream().
Now i get some exceptions and no files are downloaded.
This is the part of the code:
@Override public void run() {
try {
new File(destDir).mkdirs();
final File outFile =new File(destDir,ftpFile.getName());
final FileOutputStream fos =new FileOutputStream(outFile);
final InputStream inputStream =client.retrieveFileStream(sourceDir +ftpFile.getName());
byte b[] =new byte[4096];//todo: would be nice to have such stuff in the appl conf so users can
config it long bytesWritten =0;
while (inputStream.read(b) != -1) {
fos.write(b);
bytesWritten += b.length;
informDetailListeners(sourceDir +ftpFile.getName(), bytesWritten);
bytesTransferedTotal += b.length;
informListeners(bytesTransferedTotal);
}
client.completePendingCommand();
informListenersWeFinishedOneFile(sourceDir +ftpFile.getName(), bytesWritten);
fos.flush();
fos.close();
System.out.println(client.getReplyString());
}catch (Exception ex) {
System.out.println(client.getReplyString());
ex.printStackTrace();
}
}
The NullpointerException occurs in retrieveFileStream() - when i step
into it i will get to FTPClient.java where it says:
final boolean isInet6Address =getRemoteAddress()instanceof Inet6Address;
getRemoteAddress() has this:return _socket_.getInetAddress();
When i try to debug into this, it just jumps into my catch block with a NullPointerException.
When i add this method as watch, it says socket not connected - but it must be connected ?!
The initialisation of the ftp Client is here:
private void downloadTo(String destinationDir)throws IOException {
//first, create the destination directory so we can write into that final File file =new File(destinationDir +"/" +sfdlFile.getDescription());
file.mkdirs();
//set up the connection and go to the directory we want final ConnectionInfo connectionInfo =sfdlFile.getConnectionInfo();
final String sourceDir =sfdlFile.getRemoteSourceDir();
final FTPClient client =new FTPClient();
client.connect(connectionInfo.getHost(), connectionInfo.getPort());
System.out.println(client.getReplyString());
client.enterLocalPassiveMode();
System.out.println(client.getReplyString());
client.login(connectionInfo.getUsername(), connectionInfo.getPassword());
System.out.println(client.getReplyString());
client.setFileType(FTP.BINARY_FILE_TYPE);
System.out.println(client.getReplyString());
client.changeWorkingDirectory(sourceDir);
System.out.println(client.getReplyString());
//now retrieve all files, recursive final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(sfdlFile.getMaxThreads());
downloadDirectory(client, sourceDir, destinationDir +"/" +sfdlFile.getDescription(), scheduledExecutorService);
client.disconnect();
System.out.println(client.getReplyString());
//inform our downloader we finished }
The FTP Client is connected, the login worked, i get a lot of output from the Server which seems to be fine.
Can someone tell me whats wrong with that code? Is there something i did not read in handling recursive downloads of (bigger) files ?
The output in my intellij:
220 >>>>>>>>>>>>>>>>>>>>>>>>>
230 User logged in, proceed.
200 Type set to I.
250 Directory changed to /somedirectory
null
null
java.lang.NullPointerException
at org.apache.commons.net.SocketClient.getRemoteAddress(SocketClient.java:672)
at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:819)
at org.apache.commons.net.ftp.FTPClient._retrieveFileStream(FTPClient.java:1984)
at org.apache.commons.net.ftp.FTPClient.retrieveFileStream(FTPClient.java:1971)
at de.trustserv.jsfdl.app.sfdl.SFDLFileDownloader$DownloadRunnable.run(SFDLFileDownloader.java:145)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Re: NPE in getRemoteAdress
Posted by Oliver Zemann <ol...@gmail.com>.
ConnectionInfo is a JAXB Bean. I just have seen that thunderbird messed
up some lines during copy/paste
The connectionInfo is fully initialized, here is a short part of it:
public class ConnectionInfo {
@XmlElement(name = "Name")
private String name;
@XmlElement(name = "Host")
private String host;
@XmlElement(name = "Port")
private int port;
@XmlElement(name = "Username")
private String username;
@XmlElement(name = "Password")
...
The connection to the FTP Server works. The FTP Client is logged in, CWD
works, its just the transfer where i have problems.
Am 28.08.2016 um 16:55 schrieb Martin Gainty:
> where is connectionInfo initialised?
>
> -M
>
>> To: user@commons.apache.org
>> From: oliver.zemann@gmail.com
>> Subject: NPE in getRemoteAdress
>> Date: Sun, 28 Aug 2016 15:50:36 +0200
>>
>> Hi
>>
>> i am trying to get some files from a FTP Server. When i use
>> retrieveFile(), it works, but i would like to add some listeners to it
>> (how many bytes got transfered etc.) so i used retrieveFileStream().
>> Now i get some exceptions and no files are downloaded.
>>
>> This is the part of the code:
>>
>> @Override public void run() {
>>
>> try {
>> new File(destDir).mkdirs();
>> final File outFile =new File(destDir,ftpFile.getName());
>> final FileOutputStream fos =new FileOutputStream(outFile);
>> final InputStream inputStream =client.retrieveFileStream(sourceDir +ftpFile.getName());
>> byte b[] =new byte[4096];//todo: would be nice to have such stuff in the appl conf so users can
>> config it long bytesWritten =0;
>> while (inputStream.read(b) != -1) {
>> fos.write(b);
>> bytesWritten += b.length;
>> informDetailListeners(sourceDir +ftpFile.getName(), bytesWritten);
>> bytesTransferedTotal += b.length;
>> informListeners(bytesTransferedTotal);
>> }
>> client.completePendingCommand();
>> informListenersWeFinishedOneFile(sourceDir +ftpFile.getName(), bytesWritten);
>> fos.flush();
>> fos.close();
>> System.out.println(client.getReplyString());
>> }catch (Exception ex) {
>> System.out.println(client.getReplyString());
>> ex.printStackTrace();
>> }
>> }
>>
>> The NullpointerException occurs in retrieveFileStream() - when i step
>> into it i will get to FTPClient.java where it says:
>>
>> final boolean isInet6Address =getRemoteAddress()instanceof Inet6Address;
>>
>> getRemoteAddress() has this:return _socket_.getInetAddress();
>> When i try to debug into this, it just jumps into my catch block with a NullPointerException.
>> When i add this method as watch, it says socket not connected - but it must be connected ?!
>>
>> The initialisation of the ftp Client is here:
>> private void downloadTo(String destinationDir)throws IOException {
>>
>> //first, create the destination directory so we can write into that final File file =new File(destinationDir +"/" +sfdlFile.getDescription());
>> file.mkdirs();
>>
>> //set up the connection and go to the directory we want final ConnectionInfo connectionInfo =sfdlFile.getConnectionInfo();
>> final String sourceDir =sfdlFile.getRemoteSourceDir();
>>
>> final FTPClient client =new FTPClient();
>>
>> client.connect(connectionInfo.getHost(), connectionInfo.getPort());
>> System.out.println(client.getReplyString());
>> client.enterLocalPassiveMode();
>> System.out.println(client.getReplyString());
>> client.login(connectionInfo.getUsername(), connectionInfo.getPassword());
>> System.out.println(client.getReplyString());
>> client.setFileType(FTP.BINARY_FILE_TYPE);
>> System.out.println(client.getReplyString());
>> client.changeWorkingDirectory(sourceDir);
>> System.out.println(client.getReplyString());
>>
>> //now retrieve all files, recursive final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(sfdlFile.getMaxThreads());
>> downloadDirectory(client, sourceDir, destinationDir +"/" +sfdlFile.getDescription(), scheduledExecutorService);
>>
>> client.disconnect();
>> System.out.println(client.getReplyString());
>> //inform our downloader we finished }
>>
>> The FTP Client is connected, the login worked, i get a lot of output from the Server which seems to be fine.
>> Can someone tell me whats wrong with that code? Is there something i did not read in handling recursive downloads of (bigger) files ?
>>
>> The output in my intellij:
>> 220 >>>>>>>>>>>>>>>>>>>>>>>>>
>>
>> 230 User logged in, proceed.
>>
>> 200 Type set to I.
>>
>> 250 Directory changed to /somedirectory
>>
>> null
>> null
>> java.lang.NullPointerException
>> at org.apache.commons.net.SocketClient.getRemoteAddress(SocketClient.java:672)
>> at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:819)
>> at org.apache.commons.net.ftp.FTPClient._retrieveFileStream(FTPClient.java:1984)
>> at org.apache.commons.net.ftp.FTPClient.retrieveFileStream(FTPClient.java:1971)
>> at de.trustserv.jsfdl.app.sfdl.SFDLFileDownloader$DownloadRunnable.run(SFDLFileDownloader.java:145)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>
RE: NPE in getRemoteAdress
Posted by Martin Gainty <mg...@hotmail.com>.
where is connectionInfo initialised?
-M
> To: user@commons.apache.org
> From: oliver.zemann@gmail.com
> Subject: NPE in getRemoteAdress
> Date: Sun, 28 Aug 2016 15:50:36 +0200
>
> Hi
>
> i am trying to get some files from a FTP Server. When i use
> retrieveFile(), it works, but i would like to add some listeners to it
> (how many bytes got transfered etc.) so i used retrieveFileStream().
> Now i get some exceptions and no files are downloaded.
>
> This is the part of the code:
>
> @Override public void run() {
>
> try {
> new File(destDir).mkdirs();
> final File outFile =new File(destDir,ftpFile.getName());
> final FileOutputStream fos =new FileOutputStream(outFile);
> final InputStream inputStream =client.retrieveFileStream(sourceDir +ftpFile.getName());
> byte b[] =new byte[4096];//todo: would be nice to have such stuff in the appl conf so users can
> config it long bytesWritten =0;
> while (inputStream.read(b) != -1) {
> fos.write(b);
> bytesWritten += b.length;
> informDetailListeners(sourceDir +ftpFile.getName(), bytesWritten);
> bytesTransferedTotal += b.length;
> informListeners(bytesTransferedTotal);
> }
> client.completePendingCommand();
> informListenersWeFinishedOneFile(sourceDir +ftpFile.getName(), bytesWritten);
> fos.flush();
> fos.close();
> System.out.println(client.getReplyString());
> }catch (Exception ex) {
> System.out.println(client.getReplyString());
> ex.printStackTrace();
> }
> }
>
> The NullpointerException occurs in retrieveFileStream() - when i step
> into it i will get to FTPClient.java where it says:
>
> final boolean isInet6Address =getRemoteAddress()instanceof Inet6Address;
>
> getRemoteAddress() has this:return _socket_.getInetAddress();
> When i try to debug into this, it just jumps into my catch block with a NullPointerException.
> When i add this method as watch, it says socket not connected - but it must be connected ?!
>
> The initialisation of the ftp Client is here:
> private void downloadTo(String destinationDir)throws IOException {
>
> //first, create the destination directory so we can write into that final File file =new File(destinationDir +"/" +sfdlFile.getDescription());
> file.mkdirs();
>
> //set up the connection and go to the directory we want final ConnectionInfo connectionInfo =sfdlFile.getConnectionInfo();
> final String sourceDir =sfdlFile.getRemoteSourceDir();
>
> final FTPClient client =new FTPClient();
>
> client.connect(connectionInfo.getHost(), connectionInfo.getPort());
> System.out.println(client.getReplyString());
> client.enterLocalPassiveMode();
> System.out.println(client.getReplyString());
> client.login(connectionInfo.getUsername(), connectionInfo.getPassword());
> System.out.println(client.getReplyString());
> client.setFileType(FTP.BINARY_FILE_TYPE);
> System.out.println(client.getReplyString());
> client.changeWorkingDirectory(sourceDir);
> System.out.println(client.getReplyString());
>
> //now retrieve all files, recursive final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(sfdlFile.getMaxThreads());
> downloadDirectory(client, sourceDir, destinationDir +"/" +sfdlFile.getDescription(), scheduledExecutorService);
>
> client.disconnect();
> System.out.println(client.getReplyString());
> //inform our downloader we finished }
>
> The FTP Client is connected, the login worked, i get a lot of output from the Server which seems to be fine.
> Can someone tell me whats wrong with that code? Is there something i did not read in handling recursive downloads of (bigger) files ?
>
> The output in my intellij:
> 220 >>>>>>>>>>>>>>>>>>>>>>>>>
>
> 230 User logged in, proceed.
>
> 200 Type set to I.
>
> 250 Directory changed to /somedirectory
>
> null
> null
> java.lang.NullPointerException
> at org.apache.commons.net.SocketClient.getRemoteAddress(SocketClient.java:672)
> at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:819)
> at org.apache.commons.net.ftp.FTPClient._retrieveFileStream(FTPClient.java:1984)
> at org.apache.commons.net.ftp.FTPClient.retrieveFileStream(FTPClient.java:1971)
> at de.trustserv.jsfdl.app.sfdl.SFDLFileDownloader$DownloadRunnable.run(SFDLFileDownloader.java:145)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
Re: NPE in getRemoteAdress
Posted by Oliver Zemann <ol...@gmail.com>.
Thanks Bernd, i will fix that.
Martin, i tried the following:
client.setDataTimeout(20000);
client.setControlKeepAliveReplyTimeout(20000);
client.setConnectTimeout(20000);
Did not help, unfortunately.
Regards
Oli
Am 29.08.2016 um 01:40 schrieb Martin Gainty:
> possible timeout waiting for FTP to reply
> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w parameter
> if (args[base].equals("-w")) { controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec reply timeout
> ?
> Martin
> ______________________________________________
>
>
>
>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>> From: ecki@zusammenkunft.net
>> To: user@commons.apache.org
>> CC: oliver.zemann@gmail.com
>> Subject: Re: NPE in getRemoteAdress
>>
>> Hello,
>>
>> I am not sure about your NPE, but this code here ignores the result of
>> the read call. It cannot deal with short reads:
>>
>> Am Sun, 28 Aug 2016
>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
>>> byte b[] =new byte[4096];
>>> while (inputStream.read(b) != -1) {
>>> fos.write(b);
>>> bytesWritten += b.length;
>>
>> Gruss
>> Bernd
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>
Re: NPE in getRemoteAdress
Posted by sebb <se...@gmail.com>.
FTPClient instances are independent; each can be used by a separate thread.
However note that the instances are not inherently thread-safe.
So either create the instance in the same thread that does the download.
Alternatively, create the client before creating the thread, and
ensure that the client is only used by the new thread once created.
On 30 August 2016 at 20:53, Oliver Zemann <ol...@gmail.com> wrote:
> Seems like that made it. I used a new FTPClient for each file to download,
> that worked now in the JUnit. Will try it this weekend to get the same in my
> application. Thanks!
>
>
> Am 30.08.2016 um 16:23 schrieb sebb:
>>
>> On 29 August 2016 at 18:28, Oliver Zemann <ol...@gmail.com> wrote:
>>>
>>> My download method looks like this:
>>>
>>> private void downloadDirectory(FTPClient client, String sourceDir, String
>>> destDir,
>>> ScheduledExecutorService
>>> scheduledExecutorService) throws IOException {
>>>
>>> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
>>> for (final FTPFile ftpFile : ftpFiles) {
>>>
>>> //skip . and ..
>>> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".")
>>> ||
>>> ftpFile.getName().equals(".."))) {
>>> continue;
>>> }
>>>
>>> if (ftpFile.isDirectory()) {
>>> downloadDirectory(client, sourceDir + ftpFile.getName(),
>>> destDir + "/" + ftpFile.getName(), scheduledExecutorService);
>>> } else {
>>> // scheduledExecutorService.submit(new
>>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>>> FileOutputStream fos = new
>>> FileOutputStream(ftpFile.getName());
>>> client.retrieveFile(sourceDir + ftpFile.getName(), fos
>>> );
>>> fos.flush();
>>> fos.close();
>>> }
>>> }
>>> }
>>>
>>> When i run it like this, it works and it downloads the files. But when i
>>> use
>>> the scheduledExecuterService, it fails.
>>> So i replaced the 2 lines with my download method (where it uses the
>>> inputstream) to see what happens:
>>>
>>> } else {
>>> // scheduledExecutorService.submit(new
>>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>>> new File(destDir).mkdirs();
>>> final File outFile = new File(destDir,
>>> ftpFile.getName());
>>> final FileOutputStream fos = new
>>> FileOutputStream(outFile);
>>> final InputStream inputStream =
>>> client.retrieveFileStream(sourceDir + ftpFile.getName());
>>>
>>> byte[] buffer = new byte[8096];
>>> int len = -1;
>>> try {
>>> len = inputStream.read(buffer, 0, buffer.length);
>>> } catch (Exception ex) {
>>> ex.printStackTrace();
>>> }
>>> while (len != -1) {
>>> fos.write(buffer, 0, len);
>>> len = inputStream.read(buffer);
>>> if (Thread.interrupted()) {
>>> throw new InterruptedException();
>>> }
>>> }
>>> client.completePendingCommand();
>>> // informListenersWeFinishedOneFile(sourceDir +
>>> ftpFile.getName(), bytesWritten);
>>> fos.flush();
>>> fos.close();
>>> }
>>> }
>>>
>>> That works too?!
>>>
>>> I will go deeper into that at weekend and check whats going on, but at
>>> the
>>> moment i really dont understand it. Maybe some kind of concurrency
>>> problem
>>> with FTPClient?
>>
>> That seems the most likely.
>>
>> FTPClient is not guaranteed to be thread-safe.
>> I'm not sure that the java.io classes such as InputStream are thread
>> safe either.
>>
>>> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
>>>>
>>>> possible timeout waiting for FTP to reply
>>>> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
>>>> parameter
>>>> if (args[base].equals("-w")) {
>>>> controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]);
>>>> }
>>>> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2
>>>> sec
>>>> reply timeout
>>>> ?
>>>> Martin
>>>> ______________________________________________
>>>>
>>>>
>>>>
>>>>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>>>>> From: ecki@zusammenkunft.net
>>>>> To: user@commons.apache.org
>>>>> CC: oliver.zemann@gmail.com
>>>>> Subject: Re: NPE in getRemoteAdress
>>>>>
>>>>> Hello,
>>>>>
>>>>> I am not sure about your NPE, but this code here ignores the result of
>>>>> the read call. It cannot deal with short reads:
>>>>>
>>>>> Am Sun, 28 Aug 2016
>>>>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
>>>>>>
>>>>>> byte b[] =new byte[4096];
>>>>>> while (inputStream.read(b) != -1) {
>>>>>> fos.write(b);
>>>>>> bytesWritten += b.length;
>>>>>
>>>>>
>>>>> Gruss
>>>>> Bernd
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>>>> For additional commands, e-mail: user-help@commons.apache.org
>>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>> For additional commands, e-mail: user-help@commons.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: NPE in getRemoteAdress
Posted by Oliver Zemann <ol...@gmail.com>.
Seems like that made it. I used a new FTPClient for each file to
download, that worked now in the JUnit. Will try it this weekend to get
the same in my application. Thanks!
Am 30.08.2016 um 16:23 schrieb sebb:
> On 29 August 2016 at 18:28, Oliver Zemann <ol...@gmail.com> wrote:
>> My download method looks like this:
>>
>> private void downloadDirectory(FTPClient client, String sourceDir, String
>> destDir,
>> ScheduledExecutorService
>> scheduledExecutorService) throws IOException {
>>
>> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
>> for (final FTPFile ftpFile : ftpFiles) {
>>
>> //skip . and ..
>> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") ||
>> ftpFile.getName().equals(".."))) {
>> continue;
>> }
>>
>> if (ftpFile.isDirectory()) {
>> downloadDirectory(client, sourceDir + ftpFile.getName(),
>> destDir + "/" + ftpFile.getName(), scheduledExecutorService);
>> } else {
>> // scheduledExecutorService.submit(new
>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>> FileOutputStream fos = new
>> FileOutputStream(ftpFile.getName());
>> client.retrieveFile(sourceDir + ftpFile.getName(), fos );
>> fos.flush();
>> fos.close();
>> }
>> }
>> }
>>
>> When i run it like this, it works and it downloads the files. But when i use
>> the scheduledExecuterService, it fails.
>> So i replaced the 2 lines with my download method (where it uses the
>> inputstream) to see what happens:
>>
>> } else {
>> // scheduledExecutorService.submit(new
>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>> new File(destDir).mkdirs();
>> final File outFile = new File(destDir, ftpFile.getName());
>> final FileOutputStream fos = new FileOutputStream(outFile);
>> final InputStream inputStream =
>> client.retrieveFileStream(sourceDir + ftpFile.getName());
>>
>> byte[] buffer = new byte[8096];
>> int len = -1;
>> try {
>> len = inputStream.read(buffer, 0, buffer.length);
>> } catch (Exception ex) {
>> ex.printStackTrace();
>> }
>> while (len != -1) {
>> fos.write(buffer, 0, len);
>> len = inputStream.read(buffer);
>> if (Thread.interrupted()) {
>> throw new InterruptedException();
>> }
>> }
>> client.completePendingCommand();
>> // informListenersWeFinishedOneFile(sourceDir +
>> ftpFile.getName(), bytesWritten);
>> fos.flush();
>> fos.close();
>> }
>> }
>>
>> That works too?!
>>
>> I will go deeper into that at weekend and check whats going on, but at the
>> moment i really dont understand it. Maybe some kind of concurrency problem
>> with FTPClient?
> That seems the most likely.
>
> FTPClient is not guaranteed to be thread-safe.
> I'm not sure that the java.io classes such as InputStream are thread
> safe either.
>
>> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
>>> possible timeout waiting for FTP to reply
>>> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
>>> parameter
>>> if (args[base].equals("-w")) {
>>> controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
>>> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec
>>> reply timeout
>>> ?
>>> Martin
>>> ______________________________________________
>>>
>>>
>>>
>>>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>>>> From: ecki@zusammenkunft.net
>>>> To: user@commons.apache.org
>>>> CC: oliver.zemann@gmail.com
>>>> Subject: Re: NPE in getRemoteAdress
>>>>
>>>> Hello,
>>>>
>>>> I am not sure about your NPE, but this code here ignores the result of
>>>> the read call. It cannot deal with short reads:
>>>>
>>>> Am Sun, 28 Aug 2016
>>>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
>>>>> byte b[] =new byte[4096];
>>>>> while (inputStream.read(b) != -1) {
>>>>> fos.write(b);
>>>>> bytesWritten += b.length;
>>>>
>>>> Gruss
>>>> Bernd
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>>> For additional commands, e-mail: user-help@commons.apache.org
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: NPE in getRemoteAdress
Posted by Oliver Zemann <ol...@gmail.com>.
I wrote a simple JUnit, did not stick deeper into it but at the first
moment, it behaves just like at my application. So i *guess* its the
same problem here:
package de.mydomain.jsfdl.app.sfdl;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPFile;
import org.junit.Test;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
public class SFDLFileDownloaderTest {
private final Stringhostname ="ftp.rz.uni-wuerzburg.de";
private final int port = FTP.DEFAULT_PORT;
private final Stringusername ="anonymous";
private final Stringpassword ="blah@somemail.de";
private final StringdirectoryOfInterest ="/pub/MIRROR/kde/4.14.0/src/";
private final StringlocalDestinationDir ="/home/oli/Downloads/test";
@Test public void testFTP()throws Exception {
final FTPClient ftpClient =new FTPClient();
ftpClient.connect(hostname,port);
System.out.println(ftpClient.getReplyString());
ftpClient.login(username,password);
ftpClient.enterLocalPassiveMode();
System.out.println(ftpClient.getReplyString());
ftpClient.setFileTransferMode(FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE);
System.out.println(ftpClient.getReplyString());
ftpClient.cwd(directoryOfInterest);
System.out.println(ftpClient.getReplyString());
FTPFile files[] = ftpClient.listFiles();
System.out.println(ftpClient.getReplyString());
for(FTPFile ftpFile : files) {
final ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(5);
scheduledExecutorService.submit(new DownloadRunnable(localDestinationDir, ftpFile,directoryOfInterest, ftpClient));
}
}
private class DownloadRunnableimplements Runnable {
private final StringdestDir;
private final FTPFileftpFile;
private final StringsourceDir;
private final FTPClientclient;
public DownloadRunnable(String destDir, FTPFile ftpFile, String sourceDir, FTPClient client) {
this.destDir = destDir;
this.ftpFile = ftpFile;
this.sourceDir = sourceDir;
this.client = client;
}
@Override public void run() {
try {
new File(destDir).mkdirs();
final File outFile =new File(destDir,ftpFile.getName());
final FileOutputStream fos =new FileOutputStream(outFile);
final InputStream inputStream =client.retrieveFileStream(sourceDir +ftpFile.getName());
byte b[] =new byte[4096];
int len;
while ((len = inputStream.read(b)) != -1) {
fos.write(b,0, len);
}
client.completePendingCommand();
fos.flush();
fos.close();
System.out.println(client.getReplyString());
}catch (Exception ex) {
System.out.println(client.getReplyString());
ex.printStackTrace();
}
}
}
}
I just read http://wiki.apache.org/commons/Net/FrequentlyAskedQuestions
*Q: Are the Commons-Net classes thread-safe? For example, can you have
multiple instances of FTPClient, each with its own thread running inside?*
*A:*Multiple instances of FTPClient can be used to connect to multiple
(or the same) FTP serverand concurrently retrieve files. If you want to
share a single FTPClient instance between multiplethreads, you must
serialize access to the object with critical sections.
Can someone tell me what is meant by "critical sections" ?
Am 30.08.2016 um 16:23 schrieb sebb:
> On 29 August 2016 at 18:28, Oliver Zemann <ol...@gmail.com> wrote:
>> My download method looks like this:
>>
>> private void downloadDirectory(FTPClient client, String sourceDir, String
>> destDir,
>> ScheduledExecutorService
>> scheduledExecutorService) throws IOException {
>>
>> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
>> for (final FTPFile ftpFile : ftpFiles) {
>>
>> //skip . and ..
>> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") ||
>> ftpFile.getName().equals(".."))) {
>> continue;
>> }
>>
>> if (ftpFile.isDirectory()) {
>> downloadDirectory(client, sourceDir + ftpFile.getName(),
>> destDir + "/" + ftpFile.getName(), scheduledExecutorService);
>> } else {
>> // scheduledExecutorService.submit(new
>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>> FileOutputStream fos = new
>> FileOutputStream(ftpFile.getName());
>> client.retrieveFile(sourceDir + ftpFile.getName(), fos );
>> fos.flush();
>> fos.close();
>> }
>> }
>> }
>>
>> When i run it like this, it works and it downloads the files. But when i use
>> the scheduledExecuterService, it fails.
>> So i replaced the 2 lines with my download method (where it uses the
>> inputstream) to see what happens:
>>
>> } else {
>> // scheduledExecutorService.submit(new
>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>> new File(destDir).mkdirs();
>> final File outFile = new File(destDir, ftpFile.getName());
>> final FileOutputStream fos = new FileOutputStream(outFile);
>> final InputStream inputStream =
>> client.retrieveFileStream(sourceDir + ftpFile.getName());
>>
>> byte[] buffer = new byte[8096];
>> int len = -1;
>> try {
>> len = inputStream.read(buffer, 0, buffer.length);
>> } catch (Exception ex) {
>> ex.printStackTrace();
>> }
>> while (len != -1) {
>> fos.write(buffer, 0, len);
>> len = inputStream.read(buffer);
>> if (Thread.interrupted()) {
>> throw new InterruptedException();
>> }
>> }
>> client.completePendingCommand();
>> // informListenersWeFinishedOneFile(sourceDir +
>> ftpFile.getName(), bytesWritten);
>> fos.flush();
>> fos.close();
>> }
>> }
>>
>> That works too?!
>>
>> I will go deeper into that at weekend and check whats going on, but at the
>> moment i really dont understand it. Maybe some kind of concurrency problem
>> with FTPClient?
> That seems the most likely.
>
> FTPClient is not guaranteed to be thread-safe.
> I'm not sure that the java.io classes such as InputStream are thread
> safe either.
>
>> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
>>> possible timeout waiting for FTP to reply
>>> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
>>> parameter
>>> if (args[base].equals("-w")) {
>>> controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
>>> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec
>>> reply timeout
>>> ?
>>> Martin
>>> ______________________________________________
>>>
>>>
>>>
>>>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>>>> From: ecki@zusammenkunft.net
>>>> To: user@commons.apache.org
>>>> CC: oliver.zemann@gmail.com
>>>> Subject: Re: NPE in getRemoteAdress
>>>>
>>>> Hello,
>>>>
>>>> I am not sure about your NPE, but this code here ignores the result of
>>>> the read call. It cannot deal with short reads:
>>>>
>>>> Am Sun, 28 Aug 2016
>>>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
>>>>> byte b[] =new byte[4096];
>>>>> while (inputStream.read(b) != -1) {
>>>>> fos.write(b);
>>>>> bytesWritten += b.length;
>>>>
>>>> Gruss
>>>> Bernd
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>>> For additional commands, e-mail: user-help@commons.apache.org
>>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
Re: NPE in getRemoteAdress
Posted by sebb <se...@gmail.com>.
On 29 August 2016 at 18:28, Oliver Zemann <ol...@gmail.com> wrote:
> My download method looks like this:
>
> private void downloadDirectory(FTPClient client, String sourceDir, String
> destDir,
> ScheduledExecutorService
> scheduledExecutorService) throws IOException {
>
> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
> for (final FTPFile ftpFile : ftpFiles) {
>
> //skip . and ..
> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".") ||
> ftpFile.getName().equals(".."))) {
> continue;
> }
>
> if (ftpFile.isDirectory()) {
> downloadDirectory(client, sourceDir + ftpFile.getName(),
> destDir + "/" + ftpFile.getName(), scheduledExecutorService);
> } else {
> // scheduledExecutorService.submit(new
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> FileOutputStream fos = new
> FileOutputStream(ftpFile.getName());
> client.retrieveFile(sourceDir + ftpFile.getName(), fos );
> fos.flush();
> fos.close();
> }
> }
> }
>
> When i run it like this, it works and it downloads the files. But when i use
> the scheduledExecuterService, it fails.
> So i replaced the 2 lines with my download method (where it uses the
> inputstream) to see what happens:
>
> } else {
> // scheduledExecutorService.submit(new
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> new File(destDir).mkdirs();
> final File outFile = new File(destDir, ftpFile.getName());
> final FileOutputStream fos = new FileOutputStream(outFile);
> final InputStream inputStream =
> client.retrieveFileStream(sourceDir + ftpFile.getName());
>
> byte[] buffer = new byte[8096];
> int len = -1;
> try {
> len = inputStream.read(buffer, 0, buffer.length);
> } catch (Exception ex) {
> ex.printStackTrace();
> }
> while (len != -1) {
> fos.write(buffer, 0, len);
> len = inputStream.read(buffer);
> if (Thread.interrupted()) {
> throw new InterruptedException();
> }
> }
> client.completePendingCommand();
> // informListenersWeFinishedOneFile(sourceDir +
> ftpFile.getName(), bytesWritten);
> fos.flush();
> fos.close();
> }
> }
>
> That works too?!
>
> I will go deeper into that at weekend and check whats going on, but at the
> moment i really dont understand it. Maybe some kind of concurrency problem
> with FTPClient?
That seems the most likely.
FTPClient is not guaranteed to be thread-safe.
I'm not sure that the java.io classes such as InputStream are thread
safe either.
>
> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
>>
>> possible timeout waiting for FTP to reply
>> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w
>> parameter
>> if (args[base].equals("-w")) {
>> controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
>> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec
>> reply timeout
>> ?
>> Martin
>> ______________________________________________
>>
>>
>>
>>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>>> From: ecki@zusammenkunft.net
>>> To: user@commons.apache.org
>>> CC: oliver.zemann@gmail.com
>>> Subject: Re: NPE in getRemoteAdress
>>>
>>> Hello,
>>>
>>> I am not sure about your NPE, but this code here ignores the result of
>>> the read call. It cannot deal with short reads:
>>>
>>> Am Sun, 28 Aug 2016
>>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
>>>>
>>>> byte b[] =new byte[4096];
>>>> while (inputStream.read(b) != -1) {
>>>> fos.write(b);
>>>> bytesWritten += b.length;
>>>
>>>
>>> Gruss
>>> Bernd
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>> For additional commands, e-mail: user-help@commons.apache.org
>>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: NPE in getRemoteAdress
Posted by Martin Gainty <mg...@hotmail.com>.
if socket can stay open for normal read then that should work for filters or scheduled tasks
can you put project on github or any accessible URL and send me a link or plain attach a zip and I'll take a look
Thanks
Martin
______________________________________________
> Subject: Re: NPE in getRemoteAdress
> To: user@commons.apache.org
> From: oliver.zemann@gmail.com
> Date: Mon, 29 Aug 2016 21:27:46 +0200
>
> Hi Martin
>
> The ScheduledExecutorService is used in my project and is part of the
> JDK/JRE since version 7. This class is probably not in apache commons
> net ftp, i dont know how you got that information? Guess a misunderstanding.
>
> I read the documentation. I set the timeout of the ftp client to 20
> seconds which should be long enough. Also, i see when it jumps inside my
> threads and executes them, and here is the problem. The socket is
> closed, which should _not_ be closed. But i dont know what is causing
> this wierd behaviour.
>
> The project is open source. If you would like to take a look at it, i
> can send you the project url by private mail.
>
> Regards
>
> Oli
>
>
> Am 29.08.2016 um 21:22 schrieb Martin Gainty:
> > having difficulty locating ScheduledExecutorService in commons-nethttps://commons.apache.org/proper/commons-net/javadocs/api-3.5/index.htmlwhich version commons-net are you implementing?
> >
> > concerning ScheduledExecutorService running scheduleAtFixedRate or scheduleWithFixed methods there is a initial delay
> > built in so you maybe experiencing time-out after the initial connect during that initial delay
> > since we havent see the code you may want to look at ScheduledExecutorService examples herehttps://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledExecutorService.html
> > Martin
> >> Subject: Re: NPE in getRemoteAdress
> >> To: user@commons.apache.org
> >> From: oliver.zemann@gmail.com
> >> Date: Mon, 29 Aug 2016 19:28:56 +0200
> >>
> >> My download method looks like this:
> >>
> >> private void downloadDirectory(FTPClient client, String sourceDir,
> >> String destDir,
> >> ScheduledExecutorService
> >> scheduledExecutorService) throws IOException {
> >>
> >> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
> >> for (final FTPFile ftpFile : ftpFiles) {
> >>
> >> //skip . and ..
> >> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".")
> >> || ftpFile.getName().equals(".."))) {
> >> continue;
> >> }
> >>
> >> if (ftpFile.isDirectory()) {
> >> downloadDirectory(client, sourceDir +
> >> ftpFile.getName(), destDir + "/" + ftpFile.getName(),
> >> scheduledExecutorService);
> >> } else {
> >> // scheduledExecutorService.submit(new
> >> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> >> FileOutputStream fos = new
> >> FileOutputStream(ftpFile.getName());
> >> client.retrieveFile(sourceDir + ftpFile.getName(), fos );
> >> fos.flush();
> >> fos.close();
> >> }
> >> }
> >> }
> >>
> >> When i run it like this, it works and it downloads the files. But when i
> >> use the scheduledExecuterService, it fails.
> >> So i replaced the 2 lines with my download method (where it uses the
> >> inputstream) to see what happens:
> >>
> >> } else {
> >> // scheduledExecutorService.submit(new
> >> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> >> new File(destDir).mkdirs();
> >> final File outFile = new File(destDir, ftpFile.getName());
> >> final FileOutputStream fos = new FileOutputStream(outFile);
> >> final InputStream inputStream =
> >> client.retrieveFileStream(sourceDir + ftpFile.getName());
> >>
> >> byte[] buffer = new byte[8096];
> >> int len = -1;
> >> try {
> >> len = inputStream.read(buffer, 0, buffer.length);
> >> } catch (Exception ex) {
> >> ex.printStackTrace();
> >> }
> >> while (len != -1) {
> >> fos.write(buffer, 0, len);
> >> len = inputStream.read(buffer);
> >> if (Thread.interrupted()) {
> >> throw new InterruptedException();
> >> }
> >> }
> >> client.completePendingCommand();
> >> // informListenersWeFinishedOneFile(sourceDir +
> >> ftpFile.getName(), bytesWritten);
> >> fos.flush();
> >> fos.close();
> >> }
> >> }
> >>
> >> That works too?!
> >>
> >> I will go deeper into that at weekend and check whats going on, but at
> >> the moment i really dont understand it. Maybe some kind of concurrency
> >> problem with FTPClient?
> >>
> >>
> >> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
> >>> possible timeout waiting for FTP to reply
> >>> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w parameter
> >>> if (args[base].equals("-w")) { controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
> >>> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec reply timeout
> >>> ?
> >>> Martin
> >>> ______________________________________________
> >>>
> >>>
> >>>
> >>>> Date: Sun, 28 Aug 2016 20:06:46 +0200
> >>>> From: ecki@zusammenkunft.net
> >>>> To: user@commons.apache.org
> >>>> CC: oliver.zemann@gmail.com
> >>>> Subject: Re: NPE in getRemoteAdress
> >>>>
> >>>> Hello,
> >>>>
> >>>> I am not sure about your NPE, but this code here ignores the result of
> >>>> the read call. It cannot deal with short reads:
> >>>>
> >>>> Am Sun, 28 Aug 2016
> >>>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
> >>>>> byte b[] =new byte[4096];
> >>>>> while (inputStream.read(b) != -1) {
> >>>>> fos.write(b);
> >>>>> bytesWritten += b.length;
> >>>> Gruss
> >>>> Bernd
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> >>>> For additional commands, e-mail: user-help@commons.apache.org
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> >> For additional commands, e-mail: user-help@commons.apache.org
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
Re: NPE in getRemoteAdress
Posted by Oliver Zemann <ol...@gmail.com>.
Hi Martin
The ScheduledExecutorService is used in my project and is part of the
JDK/JRE since version 7. This class is probably not in apache commons
net ftp, i dont know how you got that information? Guess a misunderstanding.
I read the documentation. I set the timeout of the ftp client to 20
seconds which should be long enough. Also, i see when it jumps inside my
threads and executes them, and here is the problem. The socket is
closed, which should _not_ be closed. But i dont know what is causing
this wierd behaviour.
The project is open source. If you would like to take a look at it, i
can send you the project url by private mail.
Regards
Oli
Am 29.08.2016 um 21:22 schrieb Martin Gainty:
> having difficulty locating ScheduledExecutorService in commons-nethttps://commons.apache.org/proper/commons-net/javadocs/api-3.5/index.htmlwhich version commons-net are you implementing?
>
> concerning ScheduledExecutorService running scheduleAtFixedRate or scheduleWithFixed methods there is a initial delay
> built in so you maybe experiencing time-out after the initial connect during that initial delay
> since we havent see the code you may want to look at ScheduledExecutorService examples herehttps://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledExecutorService.html
> Martin
>> Subject: Re: NPE in getRemoteAdress
>> To: user@commons.apache.org
>> From: oliver.zemann@gmail.com
>> Date: Mon, 29 Aug 2016 19:28:56 +0200
>>
>> My download method looks like this:
>>
>> private void downloadDirectory(FTPClient client, String sourceDir,
>> String destDir,
>> ScheduledExecutorService
>> scheduledExecutorService) throws IOException {
>>
>> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
>> for (final FTPFile ftpFile : ftpFiles) {
>>
>> //skip . and ..
>> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".")
>> || ftpFile.getName().equals(".."))) {
>> continue;
>> }
>>
>> if (ftpFile.isDirectory()) {
>> downloadDirectory(client, sourceDir +
>> ftpFile.getName(), destDir + "/" + ftpFile.getName(),
>> scheduledExecutorService);
>> } else {
>> // scheduledExecutorService.submit(new
>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>> FileOutputStream fos = new
>> FileOutputStream(ftpFile.getName());
>> client.retrieveFile(sourceDir + ftpFile.getName(), fos );
>> fos.flush();
>> fos.close();
>> }
>> }
>> }
>>
>> When i run it like this, it works and it downloads the files. But when i
>> use the scheduledExecuterService, it fails.
>> So i replaced the 2 lines with my download method (where it uses the
>> inputstream) to see what happens:
>>
>> } else {
>> // scheduledExecutorService.submit(new
>> DownloadRunnable(destDir, ftpFile, sourceDir, client));
>> new File(destDir).mkdirs();
>> final File outFile = new File(destDir, ftpFile.getName());
>> final FileOutputStream fos = new FileOutputStream(outFile);
>> final InputStream inputStream =
>> client.retrieveFileStream(sourceDir + ftpFile.getName());
>>
>> byte[] buffer = new byte[8096];
>> int len = -1;
>> try {
>> len = inputStream.read(buffer, 0, buffer.length);
>> } catch (Exception ex) {
>> ex.printStackTrace();
>> }
>> while (len != -1) {
>> fos.write(buffer, 0, len);
>> len = inputStream.read(buffer);
>> if (Thread.interrupted()) {
>> throw new InterruptedException();
>> }
>> }
>> client.completePendingCommand();
>> // informListenersWeFinishedOneFile(sourceDir +
>> ftpFile.getName(), bytesWritten);
>> fos.flush();
>> fos.close();
>> }
>> }
>>
>> That works too?!
>>
>> I will go deeper into that at weekend and check whats going on, but at
>> the moment i really dont understand it. Maybe some kind of concurrency
>> problem with FTPClient?
>>
>>
>> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
>>> possible timeout waiting for FTP to reply
>>> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w parameter
>>> if (args[base].equals("-w")) { controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
>>> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec reply timeout
>>> ?
>>> Martin
>>> ______________________________________________
>>>
>>>
>>>
>>>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>>>> From: ecki@zusammenkunft.net
>>>> To: user@commons.apache.org
>>>> CC: oliver.zemann@gmail.com
>>>> Subject: Re: NPE in getRemoteAdress
>>>>
>>>> Hello,
>>>>
>>>> I am not sure about your NPE, but this code here ignores the result of
>>>> the read call. It cannot deal with short reads:
>>>>
>>>> Am Sun, 28 Aug 2016
>>>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
>>>>> byte b[] =new byte[4096];
>>>>> while (inputStream.read(b) != -1) {
>>>>> fos.write(b);
>>>>> bytesWritten += b.length;
>>>> Gruss
>>>> Bernd
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>>> For additional commands, e-mail: user-help@commons.apache.org
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: NPE in getRemoteAdress
Posted by Martin Gainty <mg...@hotmail.com>.
having difficulty locating ScheduledExecutorService in commons-nethttps://commons.apache.org/proper/commons-net/javadocs/api-3.5/index.htmlwhich version commons-net are you implementing?
concerning ScheduledExecutorService running scheduleAtFixedRate or scheduleWithFixed methods there is a initial delay
built in so you maybe experiencing time-out after the initial connect during that initial delay
since we havent see the code you may want to look at ScheduledExecutorService examples herehttps://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledExecutorService.html
Martin
> Subject: Re: NPE in getRemoteAdress
> To: user@commons.apache.org
> From: oliver.zemann@gmail.com
> Date: Mon, 29 Aug 2016 19:28:56 +0200
>
> My download method looks like this:
>
> private void downloadDirectory(FTPClient client, String sourceDir,
> String destDir,
> ScheduledExecutorService
> scheduledExecutorService) throws IOException {
>
> final FTPFile[] ftpFiles = client.listFiles(sourceDir);
> for (final FTPFile ftpFile : ftpFiles) {
>
> //skip . and ..
> if (ftpFile.isDirectory() && (ftpFile.getName().equals(".")
> || ftpFile.getName().equals(".."))) {
> continue;
> }
>
> if (ftpFile.isDirectory()) {
> downloadDirectory(client, sourceDir +
> ftpFile.getName(), destDir + "/" + ftpFile.getName(),
> scheduledExecutorService);
> } else {
> // scheduledExecutorService.submit(new
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> FileOutputStream fos = new
> FileOutputStream(ftpFile.getName());
> client.retrieveFile(sourceDir + ftpFile.getName(), fos );
> fos.flush();
> fos.close();
> }
> }
> }
>
> When i run it like this, it works and it downloads the files. But when i
> use the scheduledExecuterService, it fails.
> So i replaced the 2 lines with my download method (where it uses the
> inputstream) to see what happens:
>
> } else {
> // scheduledExecutorService.submit(new
> DownloadRunnable(destDir, ftpFile, sourceDir, client));
> new File(destDir).mkdirs();
> final File outFile = new File(destDir, ftpFile.getName());
> final FileOutputStream fos = new FileOutputStream(outFile);
> final InputStream inputStream =
> client.retrieveFileStream(sourceDir + ftpFile.getName());
>
> byte[] buffer = new byte[8096];
> int len = -1;
> try {
> len = inputStream.read(buffer, 0, buffer.length);
> } catch (Exception ex) {
> ex.printStackTrace();
> }
> while (len != -1) {
> fos.write(buffer, 0, len);
> len = inputStream.read(buffer);
> if (Thread.interrupted()) {
> throw new InterruptedException();
> }
> }
> client.completePendingCommand();
> // informListenersWeFinishedOneFile(sourceDir +
> ftpFile.getName(), bytesWritten);
> fos.flush();
> fos.close();
> }
> }
>
> That works too?!
>
> I will go deeper into that at weekend and check whats going on, but at
> the moment i really dont understand it. Maybe some kind of concurrency
> problem with FTPClient?
>
>
> Am 29.08.2016 um 01:40 schrieb Martin Gainty:
> > possible timeout waiting for FTP to reply
> > examples.ftp.FTPClientExample says to increase FTP reply timeout with -w parameter
> > if (args[base].equals("-w")) { controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
> > or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec reply timeout
> > ?
> > Martin
> > ______________________________________________
> >
> >
> >
> >> Date: Sun, 28 Aug 2016 20:06:46 +0200
> >> From: ecki@zusammenkunft.net
> >> To: user@commons.apache.org
> >> CC: oliver.zemann@gmail.com
> >> Subject: Re: NPE in getRemoteAdress
> >>
> >> Hello,
> >>
> >> I am not sure about your NPE, but this code here ignores the result of
> >> the read call. It cannot deal with short reads:
> >>
> >> Am Sun, 28 Aug 2016
> >> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
> >>> byte b[] =new byte[4096];
> >>> while (inputStream.read(b) != -1) {
> >>> fos.write(b);
> >>> bytesWritten += b.length;
> >>
> >> Gruss
> >> Bernd
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> >> For additional commands, e-mail: user-help@commons.apache.org
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
Re: NPE in getRemoteAdress
Posted by Oliver Zemann <ol...@gmail.com>.
My download method looks like this:
private void downloadDirectory(FTPClient client, String sourceDir,
String destDir,
ScheduledExecutorService
scheduledExecutorService) throws IOException {
final FTPFile[] ftpFiles = client.listFiles(sourceDir);
for (final FTPFile ftpFile : ftpFiles) {
//skip . and ..
if (ftpFile.isDirectory() && (ftpFile.getName().equals(".")
|| ftpFile.getName().equals(".."))) {
continue;
}
if (ftpFile.isDirectory()) {
downloadDirectory(client, sourceDir +
ftpFile.getName(), destDir + "/" + ftpFile.getName(),
scheduledExecutorService);
} else {
// scheduledExecutorService.submit(new
DownloadRunnable(destDir, ftpFile, sourceDir, client));
FileOutputStream fos = new
FileOutputStream(ftpFile.getName());
client.retrieveFile(sourceDir + ftpFile.getName(), fos );
fos.flush();
fos.close();
}
}
}
When i run it like this, it works and it downloads the files. But when i
use the scheduledExecuterService, it fails.
So i replaced the 2 lines with my download method (where it uses the
inputstream) to see what happens:
} else {
// scheduledExecutorService.submit(new
DownloadRunnable(destDir, ftpFile, sourceDir, client));
new File(destDir).mkdirs();
final File outFile = new File(destDir, ftpFile.getName());
final FileOutputStream fos = new FileOutputStream(outFile);
final InputStream inputStream =
client.retrieveFileStream(sourceDir + ftpFile.getName());
byte[] buffer = new byte[8096];
int len = -1;
try {
len = inputStream.read(buffer, 0, buffer.length);
} catch (Exception ex) {
ex.printStackTrace();
}
while (len != -1) {
fos.write(buffer, 0, len);
len = inputStream.read(buffer);
if (Thread.interrupted()) {
throw new InterruptedException();
}
}
client.completePendingCommand();
// informListenersWeFinishedOneFile(sourceDir +
ftpFile.getName(), bytesWritten);
fos.flush();
fos.close();
}
}
That works too?!
I will go deeper into that at weekend and check whats going on, but at
the moment i really dont understand it. Maybe some kind of concurrency
problem with FTPClient?
Am 29.08.2016 um 01:40 schrieb Martin Gainty:
> possible timeout waiting for FTP to reply
> examples.ftp.FTPClientExample says to increase FTP reply timeout with -w parameter
> if (args[base].equals("-w")) { controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
> or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec reply timeout
> ?
> Martin
> ______________________________________________
>
>
>
>> Date: Sun, 28 Aug 2016 20:06:46 +0200
>> From: ecki@zusammenkunft.net
>> To: user@commons.apache.org
>> CC: oliver.zemann@gmail.com
>> Subject: Re: NPE in getRemoteAdress
>>
>> Hello,
>>
>> I am not sure about your NPE, but this code here ignores the result of
>> the read call. It cannot deal with short reads:
>>
>> Am Sun, 28 Aug 2016
>> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
>>> byte b[] =new byte[4096];
>>> while (inputStream.read(b) != -1) {
>>> fos.write(b);
>>> bytesWritten += b.length;
>>
>> Gruss
>> Bernd
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
RE: NPE in getRemoteAdress
Posted by Martin Gainty <mg...@hotmail.com>.
possible timeout waiting for FTP to reply
examples.ftp.FTPClientExample says to increase FTP reply timeout with -w parameter
if (args[base].equals("-w")) { controlKeepAliveReplyTimeout = Integer.parseInt(args[++base]); }
or with FTPClient:ftpClient.setControlKeepAliveReplyTimeout(2000); //2 sec reply timeout
?
Martin
______________________________________________
> Date: Sun, 28 Aug 2016 20:06:46 +0200
> From: ecki@zusammenkunft.net
> To: user@commons.apache.org
> CC: oliver.zemann@gmail.com
> Subject: Re: NPE in getRemoteAdress
>
> Hello,
>
> I am not sure about your NPE, but this code here ignores the result of
> the read call. It cannot deal with short reads:
>
> Am Sun, 28 Aug 2016
> 15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
> > byte b[] =new byte[4096];
> > while (inputStream.read(b) != -1) {
> > fos.write(b);
> > bytesWritten += b.length;
>
>
> Gruss
> Bernd
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
Re: NPE in getRemoteAdress
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Hello,
I am not sure about your NPE, but this code here ignores the result of
the read call. It cannot deal with short reads:
Am Sun, 28 Aug 2016
15:50:36 +0200 schrieb Oliver Zemann <ol...@gmail.com>:
> byte b[] =new byte[4096];
> while (inputStream.read(b) != -1) {
> fos.write(b);
> bytesWritten += b.length;
Gruss
Bernd
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org