You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Vitaliy Khudenko (Jira)" <ji...@apache.org> on 2020/12/05 11:31:00 UTC
[jira] [Comment Edited] (NET-686) Most files aren't downloaded
completely from an FTP server
[ https://issues.apache.org/jira/browse/NET-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244465#comment-17244465 ]
Vitaliy Khudenko edited comment on NET-686 at 12/5/20, 11:30 AM:
-----------------------------------------------------------------
Looks like I've encountered this issue too.
My setup is an Android app as an FTPS client running on a physical device (Google Nexus, Android 8.1) or an emulator (Android 10) from Android Studio (Win 10). VPN.
On the FTPS server I have 3 zip files (1, 4 and 74 KBytes).
Small files (1 and 4 KB) are never causing troubles. However for the largest one the issue happens with ~25-33% probability.
The exact file size is 75'743 bytes. But it happens that only 75'741 bytes or 75'740 bytes is downloaded (missing just a few bytes) with no errors reported by {{FTPSClient}}.
Here is my code (Kotlin):
{code:java}
private val CONNECT_TIMEOUT_MILLIS = 120_000 // 2 minutes
private val SO_TIMEOUT_MILLIS = 120_000 // 2 minutes
private val BUFFER_SIZE = 1024 // 1 KB
val client: FTPSClient = FTPSClient("TLS", false)
client.connectTimeout = CONNECT_TIMEOUT_MILLIS
client.connect(hostname, port)
client.soTimeout = SO_TIMEOUT_MILLIS
if (!FTPReply.isPositiveCompletion(client.replyCode)) {
throw Exception("FTP server refused connection")
}
if (!client.login(username, password)) {
throw Exception("FTP server refused to authenticate $username")
}
FileOutputStream(localTargetFile).use { outputStream: FileOutputStream ->
val status = client.retrieveFile(ftpFilePath, outputStream)
outputStream.flush()
check(status) { "negative retrieveFile status" } // throws Exception if status is false, but this is always true
}
{code}
I tried to mitigate this with a retry mechanism where I would resume downloading for the number of the missing bytes only (e.g. the second attempt only fetches the last missing few bytes). But it still results in a corrupted zip file, despite the total number of bytes is correct. Looks like the first pass/attempt already brings a corrupted chunk, so resumed download does not help:
{code:java}
repeat(ATTEMPTS_TO_DOWNLOAD) {
if (localTargetFile.exists()) {
client.restartOffset = localTargetFile.length()
FileOutputStream(localTargetFile, true)
} else {
FileOutputStream(localTargetFile)
}.use { outputStream: FileOutputStream ->
val status = client.retrieveFile(ftpFilePath, outputStream)
outputStream.flush()
check(status) { "negative retrieveFile status" } // throws Exception if status is false, but this is always true
}
}{code}
The odd things I noticed (which is maybe due to FTPS server misconfig/missetup, but FTPS server setup is beyond my control):
* {{FTPClient.mlistFile()}} always returns {{null}}
* I am not able to browse the FTPS content via WinSCP
was (Author: vkhudenko):
Looks like I've encountered this issue too.
My setup is an Android app as an FTPS client running on a physical device (Google Nexus, Android 8.1) or an emulator (Android 10) from Android Studio (Win 10). VPN.
On the FTPS server I have 3 zip files (1, 4 and 74 KBytes).
Small files (1 and 4 KB) are never causing troubles. However for the largest one the issue happens with ~25-33% probability.
The exact file size is 75'743 bytes. But it happens that only 75'741 bytes or 75'740 bytes is downloaded (missing just a few bytes) with no errors reported by {{FTPSClient}}.
Here is my code (Kotlin):
{code:java}
private val CONNECT_TIMEOUT_MILLIS = 120_000 // 2 minutes
private val SO_TIMEOUT_MILLIS = 120_000 // 2 minutes
private val BUFFER_SIZE = 1024 // 1 KB
val client: FTPSClient = FTPSClient("TLS", false)
client.connectTimeout = CONNECT_TIMEOUT_MILLIS
client.connect(config.hostname, config.port)
client.soTimeout = SO_TIMEOUT_MILLIS
if (!FTPReply.isPositiveCompletion(client.replyCode)) {
throw Exception("FTP server refused connection")
}
if (!client.login(username, password)) {
throw Exception("FTP server refused to authenticate $username")
}
FileOutputStream(localTargetFile).use { outputStream: FileOutputStream ->
val status = client.retrieveFile(ftpFilePath, outputStream)
outputStream.flush()
check(status) { "negative retrieveFile status" } // throws Exception if status is false, but this is always true
}
{code}
I tried to mitigate this with a retry mechanism where I would resume downloading for the number of the missing bytes only (e.g. the second attempt only fetches the last missing few bytes). But it still results in a corrupted zip file, despite the total number of bytes is correct. Looks like the first pass/attempt already brings a corrupted chunk, so resumed download does not help:
{code:java}
repeat(ATTEMPTS_TO_DOWNLOAD) {
if (localTargetFile.exists()) {
client.restartOffset = localTargetFile.length()
FileOutputStream(localTargetFile, true)
} else {
FileOutputStream(localTargetFile)
}.use { outputStream: FileOutputStream ->
val status = client.retrieveFile(ftpFilePath, outputStream)
outputStream.flush()
check(status) { "negative retrieveFile status" } // throws Exception if status is false, but this is always true
}
}{code}
The odd things I noticed (which is maybe due to FTPS server misconfig/missetup, but FTPS server setup is beyond my control):
* {{FTPClient.mlistFile()}} always returns {{null}}
* I am not able to browse the FTPS content via WinSCP
> Most files aren't downloaded completely from an FTP server
> ----------------------------------------------------------
>
> Key: NET-686
> URL: https://issues.apache.org/jira/browse/NET-686
> Project: Commons Net
> Issue Type: Bug
> Components: FTP
> Affects Versions: 3.6, 3.7.2
> Environment: Win 10
> Java 8
> Android Studio 3.6.1 (min SDK 24, target SDK 27)
> Reporter: JRRR
> Priority: Major
> Attachments: 2a-original.png, 2b-corrupt.png, 2c-corrupt.png, 5a-original.jpg, 5b-corrupt.jpg, 5c-corrupt.jpg, DownloadProblem.java
>
>
> About a month ago I opened another [issue|https://issues.apache.org/jira/browse/NET-684] that was closed because it wasn't reproducible with macOS and a public FTP server.
> Short summary: Downloading files from an FTP server results in these files randomly missing bytes. It looks like the download always "completes" and there are no error messages/exceptions but random bytes in random files are simply skipped. Images (jpg & png) are usually affected more (up to 30, maybe 40, bytes smaller than the original), and are then also visibly corrupt, than text files (usually only 2-3 bytes smaller, rarely more).
> I'm working on an Android app (Win 10, Java 8, Android Studio 3.6.1, min SDK 24, target SDK 27), which I'm testing with FTP servers in the same network (1x Win 10, 1x Linux, both accessed via IP - "10.1.1.xxx"). No matter what method in the library I use (retrieveFile, retrieveFileStream, sendCommand(FTPCmd.RETRIEVE, filename)), most of the time there's at least a single file that's corrupted.
> I also tested the same code with public servers and even though I didn't have a lot of time because those servers regularely delete uploaded files, I never experienced said problem with them.
> I even wrote my own mini-library (just for login/logout and download) using Java's default "Socket" but I still had the same problem on Android Studio's simulator/a real device. BUT: When I used the same code to create a small Windows/Swing/Java app, there were no more corrupted files.
> It looks like this bug is only affecting a very specific combination of OS,...:
> Android (emulator/real device) + Java (8) + FTP server in the same network (accessed via IP)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)