[jira] [Closed] (JCLOUDS-1622) Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected

["Ivan Dimitrov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/JCLOUDS-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ivan Dimitrov closed JCLOUDS-1622.
----------------------------------
    Resolution: Abandoned

> Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: JCLOUDS-1622
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1622
>             Project: jclouds
>          Issue Type: Bug
>    Affects Versions: 2.5.0
>            Reporter: Ivan Dimitrov
>            Assignee: Andrew Gaul
>            Priority: Major
>         Attachments: jclouds-wire.log, jclouds.log
>
>
> Hello,
> We are observing an issue with JClouds while transferring a file with an input stream only.
> We are opening an input stream to a remote file and we are passing the stream to the JClouds library, but after some time it fails with the following error:
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
> 	at com.google.common.base.Throwables.propagate(Throwables.java:241)
> 	at org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
> 	at org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
> 	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
> 	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
> 	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
> 	at org.example.Main.main(Main.java:50)
> Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
> 	at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
> 	at org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
> 	... 5 more
> {noformat}
> Note: The issue occurs only for larger files (more than 32 mb).
> Java version: sapmachine-jdk-11.0.15.0.1.jdk
> IAAS: AWS-s3
> If we pass a BufferedInputStream the code works, so I guess that there is some issue with the processing of the stream. The stream returned by the Java Http Client is HttpResponseInputStream.
> Code:
> {code:title=Main.java|borderStyle=solid}
> package org.example;
> import com.google.common.collect.ImmutableSet;
> import com.google.inject.Module;
> import org.jclouds.ContextBuilder;
> import org.jclouds.blobstore.BlobStore;
> import org.jclouds.blobstore.BlobStoreContext;
> import org.jclouds.blobstore.domain.Blob;
> import org.jclouds.blobstore.options.PutOptions;
> import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
> import java.io.BufferedInputStream;
> import java.io.InputStream;
> import java.net.Authenticator;
> import java.net.PasswordAuthentication;
> import java.net.URI;
> import java.net.http.HttpClient;
> import java.net.http.HttpRequest;
> import java.net.http.HttpResponse;
> import java.time.Duration;
> import java.util.UUID;
> public class Main {
>     public static void main(String[] args) throws Exception {
>         Iterable<Module> modules = ImmutableSet.<Module>of(
>                 new SLF4JLoggingModule());
>         ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
>                 .credentials("<IDENTITY>", "<CREDENTIAL>")
>                 .modules(modules);
>         BlobStoreContext blobStoreContext = contextBuilder.buildView(BlobStoreContext.class);
>         BlobStore blobStore = blobStoreContext.getBlobStore();
>         String decodedUrl = "<URL>";
>         HttpClient client = buildHttpClient(decodedUrl);
>         HttpResponse<InputStream> response = callRemoteEndpointWithRetry(client, decodedUrl);
>         long fileSize = response.headers()
>                 .firstValueAsLong("Content-Length")
>                 .orElseThrow(() -> new IllegalArgumentException("No Content-Length"));
>         System.out.println("Bytes: " + fileSize);
>         InputStream inputStream = response.body();
>             String container = "<CONTAINER>";
>             Blob blob = blobStore.blobBuilder("TEST.zip")
> //                    .payload(new BufferedInputStream(inputStream, 8 * 1024))
>                     .payload(inputStream)
>                     .contentDisposition(UUID.randomUUID().toString())
>                     .contentType("application/octet-stream")
>                     .contentLength(fileSize)
>                     .build();
>             blobStore.putBlob(container, blob, PutOptions.Builder.multipart());
>     }
>     private static HttpClient buildHttpClient(String decodedUrl) {
>         return HttpClient.newBuilder()
>                 .version(HttpClient.Version.HTTP_1_1)
>                 .connectTimeout(Duration.ofMinutes(60))
>                 .followRedirects(HttpClient.Redirect.NORMAL)
>                 .authenticator(buildPasswordAuthenticator(decodedUrl))
>                 .build();
>     }
>     private static Authenticator buildPasswordAuthenticator(String decodedUrl) {
>         return new Authenticator() {
>             @Override
>             protected PasswordAuthentication getPasswordAuthentication() {
>                 var uri = URI.create(decodedUrl);
>                 var userInfo = uri.getUserInfo();
>                 if (userInfo != null) {
>                     var separatorIndex = userInfo.indexOf(':');
>                     var username = userInfo.substring(0, separatorIndex);
>                     var password = userInfo.substring(separatorIndex + 1);
>                     return new PasswordAuthentication(username, password.toCharArray());
>                 }
>                 return super.getPasswordAuthentication();
>             }
>         };
>     }
>     private static HttpResponse<InputStream> callRemoteEndpointWithRetry(HttpClient client, String decodedUrl) throws Exception {
>         var response = client.send(buildFetchFileRequest(decodedUrl), HttpResponse.BodyHandlers.ofInputStream());
>         if (response.statusCode() / 100 != 2) {
>             throw new IllegalStateException("INVALID CODE " + response.statusCode());
>         }
>         return response;
>     }
>     private static HttpRequest buildFetchFileRequest(String decodedUrl) {
>         var builder = HttpRequest.newBuilder()
>                 .GET()
>                 .expectContinue(true)
>                 .headers("Content-Type", "multipart/form-data")
>                 .timeout(Duration.ofMinutes(30));
>         var uri = URI.create(decodedUrl);
>         var userInfo = uri.getUserInfo();
>         if (userInfo != null) {
>             builder.uri(URI.create(decodedUrl.replace(userInfo + "@", "")));
>         } else {
>             builder.uri(uri);
>         }
>         return builder.build();
>     }
> }
> {code}
> Could you please take a look? Any advice would be helpful. The one thing I see differences between the HttpResponseInputStream and the BufferedInputStream is that HttpResponseInputStream::isAvailable returns 0. But I'm not sure if that's the issue because the skip of the stream is done by just reading it.
> Note: Removed the binary content from jclouds-wire.log
> Thanks!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)