You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by jskora <gi...@git.apache.org> on 2015/11/24 18:29:09 UTC

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

GitHub user jskora opened a pull request:

    https://github.com/apache/nifi/pull/132

    NIFI-1107 - Integrate Multipart uploads into the PutS3Object processor.

    * Add nifi-ssl-context-service-nar to nifi-aws-nar pom.
    * Add nifi-ssl-context-service-api to nifi-aws-processors pom.
    * Add SSL context service and endpoint override properties to AbstractAWSProcessor to support non-Amazon S3-compatible endpoints.
    * Added Multipart upload API support, multipart threshold and size properties, state tracking for multipart upload, and expanded flow file attributes to PutS3Object.
    * Updated annotations and property list for FetchS3Object and DeleteS3Object.
    * Added region to bucket in AbstractS3Test to minimize AWS propogation issues during testing.
    * Expanded tests in TestPutS3Object to validate new multipart logic.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jskora/nifi NIFI-1107

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #132
    
----
commit 52fd4180c7e929cbbdd3647d1b7aad68174b2f9c
Author: Joe Skora <js...@gmail.com>
Date:   2015-11-24T17:15:01Z

    Integrate Multipart uploads into the PutS3Object processor.
    * Add nifi-ssl-context-service-nar to nifi-aws-nar pom.
    * Add nifi-ssl-context-service-api to nifi-aws-processors pom.
    * Add SSL context service and endpoint override properties to AbstractAWSProcessor to support non-Amazon S3-compatible endpoints.
    * Added Multipart upload API support, multipart threshold and size properties, state tracking for multipart upload, and expanded flow file attributes to PutS3Object.
    * Updated annotations and property list for FetchS3Object and DeleteS3Object.
    * Added region to bucket in AbstractS3Test to minimize AWS propogation issues during testing.
    * Expanded tests in TestPutS3Object to validate new multipart logic.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/132#discussion_r45942649
  
    --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java ---
    @@ -102,6 +177,94 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String
                 .build();
         }
     
    +    protected File getPersistenceFile() {
    +        return new File(PERSISTENCE_ROOT + getIdentifier());
    +    }
    +
    +    @Override
    +    public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) {
    +        if (descriptor.equals(KEY)
    +                || descriptor.equals(BUCKET)
    +                || descriptor.equals(ENDPOINT_OVERRIDE)
    +                || descriptor.equals(STORAGE_CLASS)
    +                || descriptor.equals(REGION)) {
    +            destroyState();
    +        }
    +    }
    +
    +    protected MultipartState getState(final String s3ObjectKey) throws IOException {
    --- End diff --
    
    properties can only be modified if the processor is stopped so it *should be safe* i believe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by jskora <gi...@git.apache.org>.
Github user jskora commented on the pull request:

    https://github.com/apache/nifi/pull/132#issuecomment-169329852
  
    I'm closing this pull request since the general S3 changes have been incorporated by @trkurc.  The remaining Multipart Upload functionality will be incorporated into a new pull request in conjunction with S3 bucket and local state cleanup functionality.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on the pull request:

    https://github.com/apache/nifi/pull/132#issuecomment-159806672
  
    Another, major secondary concern is what to do with failed partial multipart puts and maybe having to do bucket cleanup.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #132: NIFI-1107 - Integrate Multipart uploads into the PutS3Objec...

Posted by coveralls <gi...@git.apache.org>.
Github user coveralls commented on the issue:

    https://github.com/apache/nifi/pull/132
  
    
    [![Coverage Status](https://coveralls.io/builds/13545015/badge)](https://coveralls.io/builds/13545015)
    
    Changes Unknown when pulling **52fd4180c7e929cbbdd3647d1b7aad68174b2f9c on jskora:NIFI-1107** into ** on apache:master**.



---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by jskora <gi...@git.apache.org>.
Github user jskora closed the pull request at:

    https://github.com/apache/nifi/pull/132


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/132#discussion_r45943160
  
    --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java ---
    @@ -102,6 +177,94 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String
                 .build();
         }
     
    +    protected File getPersistenceFile() {
    +        return new File(PERSISTENCE_ROOT + getIdentifier());
    +    }
    +
    +    @Override
    +    public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) {
    +        if (descriptor.equals(KEY)
    +                || descriptor.equals(BUCKET)
    +                || descriptor.equals(ENDPOINT_OVERRIDE)
    +                || descriptor.equals(STORAGE_CLASS)
    +                || descriptor.equals(REGION)) {
    +            destroyState();
    +        }
    +    }
    +
    +    protected MultipartState getState(final String s3ObjectKey) throws IOException {
    --- End diff --
    
    @joewitt  - I was talking about the *State methods not the onPropertyModified.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/132#discussion_r45942561
  
    --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java ---
    @@ -102,6 +177,94 @@ protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String
                 .build();
         }
     
    +    protected File getPersistenceFile() {
    +        return new File(PERSISTENCE_ROOT + getIdentifier());
    +    }
    +
    +    @Override
    +    public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) {
    +        if (descriptor.equals(KEY)
    +                || descriptor.equals(BUCKET)
    +                || descriptor.equals(ENDPOINT_OVERRIDE)
    +                || descriptor.equals(STORAGE_CLASS)
    +                || descriptor.equals(REGION)) {
    +            destroyState();
    +        }
    +    }
    +
    +    protected MultipartState getState(final String s3ObjectKey) throws IOException {
    --- End diff --
    
    I believe these methods could be problematic if multiple threads are monkeying with the persistence file at the same time. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on the pull request:

    https://github.com/apache/nifi/pull/132#issuecomment-159804534
  
    My major concern is with the persistence file. I would think scrubbing the file periodically, removing older stale entries would be a must, otherwise the file will begin accumulating cruft as more and more files don't complete the multipart lifecycle. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1107 - Integrate Multipart uploads into th...

Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/132#discussion_r45944159
  
    --- Diff: nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java ---
    @@ -143,23 +316,223 @@ public void process(final InputStream rawIn) throws IOException {
                                 objectMetadata.setUserMetadata(userMetadata);
                             }
     
    -                        final PutObjectRequest request = new PutObjectRequest(bucket, key, in, objectMetadata);
    -                        request.setStorageClass(StorageClass.valueOf(context.getProperty(STORAGE_CLASS).getValue()));
    -                        final AccessControlList acl = createACL(context, ff);
    -                        if (acl != null) {
    -                            request.setAccessControlList(acl);
    -                        }
    +                        if (ff.getSize() <= multipartThreshold) {
    +                            //----------------------------------------
    +                            // single part upload
    +                            //----------------------------------------
    +                            final PutObjectRequest request = new PutObjectRequest(bucket, key, in, objectMetadata);
    +                            request.setStorageClass(
    +                                    StorageClass.valueOf(context.getProperty(STORAGE_CLASS).getValue()));
    +                            final AccessControlList acl = createACL(context, ff);
    +                            if (acl != null) {
    +                                request.setAccessControlList(acl);
    +                            }
     
    -                        final PutObjectResult result = s3.putObject(request);
    -                        if (result.getVersionId() != null) {
    -                            attributes.put("s3.version", result.getVersionId());
    -                        }
    +                            try {
    +                                final PutObjectResult result = s3.putObject(request);
    +                                if (result.getVersionId() != null) {
    +                                    attributes.put(S3_VERSION_ATTR_KEY, result.getVersionId());
    +                                }
    +                                if (result.getETag() != null) {
    +                                    attributes.put(S3_ETAG_ATTR_KEY, result.getETag());
    +                                }
    +                                if (result.getExpirationTime() != null) {
    +                                    attributes.put(S3_EXPIRATION_ATTR_KEY, result.getExpirationTime().toString());
    +                                }
    +                                if (result.getMetadata().getRawMetadata().keySet().contains(S3_STORAGECLASS_META_KEY)) {
    +                                    attributes.put(S3_STORAGECLASS_ATTR_KEY,
    +                                            result.getMetadata().getRawMetadataValue(S3_STORAGECLASS_META_KEY).toString());
    +                                }
    +                                if (userMetadata.size() > 0) {
    +                                    StringBuilder userMetaBldr = new StringBuilder();
    +                                    for (String userKey : userMetadata.keySet()) {
    +                                        userMetaBldr.append(userKey).append("=").append(userMetadata.get(userKey));
    +                                    }
    +                                    attributes.put(S3_USERMETA_ATTR_KEY, userMetaBldr.toString());
    +                                }
    +                            } catch (AmazonClientException e) {
    +                                getLogger().info("Failure completing upload flowfile={} bucket={} key={} reason={}",
    +                                        new Object[]{ffFilename, bucket, key, e.getMessage()});
    +                                throw (e);
    +                            }
    +                        } else {
    +                            //----------------------------------------
    +                            // multippart upload
    +                            //----------------------------------------
     
    -                        attributes.put("s3.etag", result.getETag());
    +                            // load or create persistent state
    +                            //------------------------------------------------------------
    +                            MultipartState currentState;
    +                            try {
    +                                currentState = getState(cacheKey);
    +                                if (currentState != null) {
    +                                    if (currentState.getPartETags().size() > 0) {
    +                                        final PartETag lastETag = currentState.getPartETags().get(
    +                                                currentState.getPartETags().size() - 1);
    +                                        getLogger().info("Resuming upload for flowfile='{}' bucket='{}' key='{}' " +
    +                                                "uploadID='{}' filePosition='{}' partSize='{}' storageClass='{}' " +
    +                                                "contentLength='{}' partsLoaded={} lastPart={}/{}",
    +                                                new Object[]{ffFilename, bucket, key, currentState.getUploadId(),
    +                                                        currentState.getFilePosition(), currentState.getPartSize(),
    +                                                        currentState.getStorageClass().toString(),
    +                                                        currentState.getContentLength(),
    +                                                        currentState.getPartETags().size(),
    +                                                        Integer.toString(lastETag.getPartNumber()),
    +                                                        lastETag.getETag()});
    +                                    } else {
    +                                        getLogger().info("Resuming upload for flowfile='{}' bucket='{}' key='{}' " +
    +                                                "uploadID='{}' filePosition='{}' partSize='{}' storageClass='{}' " +
    +                                                "contentLength='{}' no partsLoaded",
    +                                                new Object[]{ffFilename, bucket, key, currentState.getUploadId(),
    +                                                        currentState.getFilePosition(), currentState.getPartSize(),
    +                                                        currentState.getStorageClass().toString(),
    +                                                        currentState.getContentLength()});
    +                                    }
    +                                } else {
    +                                    currentState = new MultipartState();
    +                                    currentState.setPartSize(multipartPartSize);
    +                                    currentState.setStorageClass(
    +                                            StorageClass.valueOf(context.getProperty(STORAGE_CLASS).getValue()));
    +                                    currentState.setContentLength(ff.getSize());
    +                                    persistState(cacheKey, currentState);
    +                                    getLogger().info("Starting new upload for flowfile='{}' bucket='{}' key='{}'",
    +                                            new Object[]{ffFilename, bucket, key});
    +                                }
    +                            } catch (IOException e) {
    +                                getLogger().error("IOException initiating cache state while processing flow files: " +
    +                                        e.getMessage());
    +                                throw (e);
    +                            }
    +
    +                            // initiate multipart upload or find position in file
    +                            //------------------------------------------------------------
    +                            if (currentState.getUploadId().isEmpty()) {
    +                                final InitiateMultipartUploadRequest initiateRequest =
    +                                        new InitiateMultipartUploadRequest(bucket, key, objectMetadata);
    +                                initiateRequest.setStorageClass(currentState.getStorageClass());
    +                                final AccessControlList acl = createACL(context, ff);
    +                                if (acl != null) {
    +                                    initiateRequest.setAccessControlList(acl);
    +                                }
    +                                try {
    +                                    final InitiateMultipartUploadResult initiateResult =
    +                                            s3.initiateMultipartUpload(initiateRequest);
    +                                    currentState.setUploadId(initiateResult.getUploadId());
    +                                    currentState.getPartETags().clear();
    +                                    try {
    +                                        persistState(cacheKey, currentState);
    +                                    } catch (Exception e) {
    +                                        getLogger().info("Exception saving cache state while processing flow file: " +
    +                                                e.getMessage());
    +                                        throw(new ProcessException("Exception saving cache state", e));
    +                                    }
    +                                    getLogger().info("Success initiating upload flowfile={} available={} position={} " +
    +                                            "length={} bucket={} key={} uploadId={}",
    +                                            new Object[]{ffFilename, in.available(), currentState.getFilePosition(),
    +                                                    currentState.getContentLength(), bucket, key,
    +                                                    currentState.getUploadId()});
    +                                    if (initiateResult.getUploadId() != null) {
    +                                        attributes.put(S3_UPLOAD_ID_ATTR_KEY, initiateResult.getUploadId());
    +                                    }
    +                                } catch (AmazonClientException e) {
    +                                    getLogger().info("Failure initiating upload flowfile={} bucket={} key={} reason={}",
    +                                            new Object[]{ffFilename, bucket, key, e.getMessage()});
    +                                    throw(e);
    +                                }
    +                            } else {
    +                                if (currentState.getFilePosition() > 0) {
    +                                    try {
    +                                        final long skipped = in.skip(currentState.getFilePosition());
    +                                        if (skipped != currentState.getFilePosition()) {
    +                                            getLogger().info("Failure skipping to resume upload flowfile={} " +
    +                                                    "bucket={} key={} position={} skipped={}",
    +                                                    new Object[]{ffFilename, bucket, key,
    +                                                            currentState.getFilePosition(), skipped});
    +                                        }
    +                                    } catch (Exception e) {
    +                                        getLogger().info("Failure skipping to resume upload flowfile={} bucket={} " +
    +                                                "key={} position={} reason={}",
    +                                                new Object[]{ffFilename, bucket, key, currentState.getFilePosition(),
    +                                                        e.getMessage()});
    +                                        throw(new ProcessException(e));
    +                                    }
    +                                }
    +                            }
    +
    +                            // upload parts
    +                            //------------------------------------------------------------
    +                            long thisPartSize;
    +                            for (int part = currentState.getPartETags().size() + 1;
    +                                 currentState.getFilePosition() < currentState.getContentLength(); part++) {
    +                                if (!PutS3Object.this.isScheduled()) {
    +                                    getLogger().info("Processor unscheduled, stopping upload flowfile={} part={} " +
    +                                            "uploadId={}", new Object[]{ffFilename, part, currentState.getUploadId()});
    +                                    session.rollback();
    +                                    return;
    +                                }
    +                                thisPartSize = Math.min(currentState.getPartSize(),
    +                                        (currentState.getContentLength() - currentState.getFilePosition()));
    +                                UploadPartRequest uploadRequest = new UploadPartRequest()
    +                                        .withBucketName(bucket)
    +                                        .withKey(key)
    +                                        .withUploadId(currentState.getUploadId())
    +                                        .withInputStream(in)
    +                                        .withPartNumber(part)
    +                                        .withPartSize(thisPartSize);
    +                                try {
    +                                    UploadPartResult uploadPartResult = s3.uploadPart(uploadRequest);
    +                                    currentState.addPartETag(uploadPartResult.getPartETag());
    +                                    currentState.setFilePosition(currentState.getFilePosition() + thisPartSize);
    +                                    try {
    +                                        persistState(cacheKey, currentState);
    +                                    } catch (Exception e) {
    --- End diff --
    
    I'd add a comment here on why swallowing exceptions here is okay, as it is not immediately intuitive.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---