You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Bernd Eckenfels (JIRA)" <ji...@apache.org> on 2017/11/03 22:09:00 UTC
[jira] [Commented] (VFS-647) calling findFiles() causes copyFrom()
to fail with a partially downloaded file.
[ https://issues.apache.org/jira/browse/VFS-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16238470#comment-16238470 ]
Bernd Eckenfels commented on VFS-647:
-------------------------------------
It looks like your two threads both close the filesystems and the fsm concurrently. You need to wait for all filesystems operations to be finished before using close.
> calling findFiles() causes copyFrom() to fail with a partially downloaded file.
> -------------------------------------------------------------------------------
>
> Key: VFS-647
> URL: https://issues.apache.org/jira/browse/VFS-647
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: 2.0, 2.2
> Environment: Windows 7 and Linux Red Hat.
> Reporter: Kenji
> Priority: Minor
>
> Edit: The problem isn't specific to findFiles. All I need to do is have 1 thread downloads the file, and the other thread just log in, have a while loop for x seconds where x < download time (using Thread.sleep(x) causes interruption to happen immediately). Once x seconds passed, the download will get interupted with pipe closed. Using the same hostname but different username and password does not cause interruption.
> Original description:
> When using FileObject.copyFrom(remote, new AllFileSelector()) to download file from remote to local directory. If SftpFileObject.findFiles(new FileDepthSelector(1,1)) is called and finished, then the copyFrom will get interrupted with error pipe closed.
> Below are test codes and stack trace error. In real scenario, this all happens within 1 or 2 second time frame. However, with test scenario, I wasn't able to reproduce it easily therefore I had to choose a file that takes around 10 seconds to download and have Thread.sleep(5000) after findFiles() call. Only tested jsch-0.1.52, jsch-0.1.54, commons-vfs2-2.0 and commons-vfs2-2.2.
> {code:java}
> public class FtpClient {
>
> public static void main(String[] args) {
> if (args.length < 5) {
> throw new RuntimeException("args: host user pass local remote");
> }
> String hostname = args[0];
> String username = args[1];
> String password = args[2];
> int port = 22;
> String local = args[3];
> String remote = args[4];
> final String remoteDir = remote.substring(0, Math.max(0, remote.lastIndexOf("/")));
> Thread t0 = new Thread() {
> public void run() {
> try (Ftp ftp = new Ftp(hostname, port)) {
> ftp.login(username, password);
> ftp.list(remoteDir);
> System.out.println("findFiles() completed.");
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> };
>
> Thread ti = new Thread() {
> public void run() {
> try (Ftp ftp = new Ftp(hostname, port)) {
> ftp.login(username, password);
> ftp.download(local, remote);
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> };
> t0.start();
> ti.start();
> }
> }
> public class Ftp implements AutoCloseable {
> private int timeout = 0;
> private StaticUserAuthenticator userAuth;
> private FileSystemOptions fileSysOpts;
> private FileObject scr = null;
> private FileSystemManager fsm = null;
> private String hostName;
> private int port = 0;
> public Ftp(String remoteHost, int controlPort) throws FileSystemException {
> hostName = remoteHost;
> port = controlPort;
> fsm = VFS.getManager();
> }
> //login into a server with a valid account
> public void login(String user, String password) throws IOException {
> userAuth = new StaticUserAuthenticator(null, user, password);
> fileSysOpts = new FileSystemOptions();
> DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(fileSysOpts, userAuth);
> SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
> SftpFileSystemConfigBuilder.getInstance().setTimeout(fileSysOpts, timeout);
> SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(fileSysOpts, false);
> scr = (FileObject)fsm.resolveFile("sftp://" + hostName, fileSysOpts);
>
> System.out.println("login successfully.");
> }
> public void list(String dirName) throws Exception {
> SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + dirName, fileSysOpts);
> FileObject[] afo = RemoteFo.findFiles(new FileDepthSelector(1,1));
> Thread.sleep(5000); //key. this must be here (or some other processing that takes time) for error to reproduce consistently.
> }
> public void download(String localPath, String remoteFile) throws FileSystemException {
> SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(fileSysOpts, "no");
> SftpFileObject RemoteFo = (SftpFileObject)fsm.resolveFile("sftp://" + hostName + remoteFile ,fileSysOpts);
> scr = fsm.resolveFile("file:" + localPath);
> scr.copyFrom(RemoteFo, new AllFileSelector());
> }
>
> @Override
> public void close() {
> try{
> FileSystem fs = null;
> if(this.scr!=null){
> this.scr.close();
> fs = this.scr.getFileSystem();
> this.fsm.closeFileSystem(fs);
> }
> }catch(Exception e){
> System.out.println("unable to release ftp connection.");
> e.printStackTrace();
> }
> }
> }
> {code}
> StackTrace:
> {noformat}
> Oct 18, 2017 12:42:51 PM org.apache.commons.vfs2.VfsLog info
> INFO: Using "C:\Users\kenji\AppData\Local\Temp\vfs_cache" as temporary files stor
> e.
> login successfully.
> login successfully.
> findFiles() completed.
> org.apache.commons.vfs2.FileSystemException: Could not copy "sftp://hostname/path/to/file/FILE1" to "file:///C:/test/ftp/FILE1".
> at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
> FileObject.java:1062)
> at Ftp.download(Ftp.java:59)
> at FtpClient$2.run(FtpClient.java:36)
> Caused by: org.apache.commons.vfs2.FileSystemException: Could not close the inpu
> t stream for file "sftp://hostname/path/to/file/FILE1".
> at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
> tream.close(DefaultFileContent.java:611)
> at org.apache.commons.vfs2.FileUtil.writeContent(FileUtil.java:95)
> at org.apache.commons.vfs2.FileUtil.copyContent(FileUtil.java:114)
> at org.apache.commons.vfs2.provider.AbstractFileObject.copyFrom(Abstract
> FileObject.java:1053)
> ... 2 more
> Caused by: java.io.IOException: Pipe closed
> at java.io.PipedInputStream.read(PipedInputStream.java:307)
> at java.io.PipedInputStream.read(PipedInputStream.java:377)
> at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2882)
> at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2908)
> at com.jcraft.jsch.ChannelSftp.access$500(ChannelSftp.java:36)
> at com.jcraft.jsch.ChannelSftp$RequestQueue.cancel(ChannelSftp.java:1238
> )
> at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1503)
> at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
> at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
> eam.java:129)
> at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
> at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStr
> eam.java:129)
> at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputS
> tream.close(DefaultFileContent.java:607)
> ... 5 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)