[jira] [Created] (JCLOUDS-1608) Slicing of large files can lead to exceed the 32 parts limit of GCS

["Jan Vermeulen (Jira)" <ji...@apache.org>]

Jan Vermeulen created JCLOUDS-1608:
--------------------------------------

             Summary: Slicing of large files can lead to exceed the 32 parts limit of GCS
                 Key: JCLOUDS-1608
                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1608
             Project: jclouds
          Issue Type: Bug
          Components: jclouds-blobstore
    Affects Versions: 2.5.0, 2.2.1
            Reporter: Jan Vermeulen
             Fix For: 2.6.0, 2.2.1


MultipartUploadSlicingAlgorithm calculates slices for a large file by first using the defaultPartSize. If the results of that slicing gives parts that don't exceed the min/maxPartSizes (5MB and 5GB for GoogleCloudStorageBlobStore) but that do exceed the maxNumberOfParts (32 for GoogleCloudStorageBlobStore), the algoritm sets the number of parts to 32 and recalculates the size of the parts. If there is any remainder after that, JClouds ends up uploading 33 parts in total to GCS, causing the process to fail in completeMultipartUpload() when recomposing the original content from the parts.

The following simple unitTest proves the case:

{{public class AlgoritmTest extends TestCase {}}
{{    public void testSlicing() {}}
{{        MultipartUploadSlicingAlgorithm algorithm = new MultipartUploadSlicingAlgorithm(1024*1024*5,1024*1024*1024*5,32);}}
{{        algorithm.calculateChunkSize(1024*1024*1200+33);}}
{{        assertTrue(algorithm.getParts()+((algorithm.getRemaining() > 0)?(1):(0)) <= 32);}}
{{    }}}
{{}}}

It simulates the slicing of a file of 1.2GB+33 bytes (to make sure there is a remainder).

The following patch fixes the issue:
{{      }}

{{      ...}}

{{      long remainder = length % unitPartSize;     }}
{{      // SHB patch}}
{{      // remainder should be distributed over parts if we are at the maximumNumberOfParts}}
{{      // (if not, an additional part is uploaded to GCS thus exceeding the maximum allowed parts)}}
{{      // if (remainder == 0 && parts > 0) {}}
{{      //     parts -= 1;}}
{{      // }}}
{{      if (remainder > 0 && parts == maximumNumberOfParts) {}}
{{          parts -= 1;}}
{{          partSize = length/parts;}}
{{      }}}
{{      // end of SHB patch
      ...}}

I also commented the code that reduces the number of parts when there is no remainder, since that ends up creating a remaining part that is the same size as the others.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)