You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/12/07 06:28:29 UTC
[GitHub] [nifi] malthe opened a new pull request, #6765: NIFI-9972 Put blob can now pull from another blob source
malthe opened a new pull request, #6765:
URL: https://github.com/apache/nifi/pull/6765
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-9972](https://issues.apache.org/jira/browse/NIFI-9972)
This adds support for the "Put Blob from URL" operation which provides service-to-service copying of blobs – the client is only responsible for the orchestration which copies individual blocks, not for transferring the data in those blocks.
The functionality is added as an optional new data source of the [PutAzureBlobStorage_v12](https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-azure-nar/1.19.0/org.apache.nifi.processors.azure.storage.PutAzureBlobStorage_v12/) processor.
The test case is not comprehensive in the sense that only the built-in credential is tested (account key). Meanwhile, all credentials are indeed supported (defined by the `AzureStorageCredentialsType` enum):
- Account key
- SAS token
- Access token
- Managed identity
- Service principal
The logic of the authentication code is that it's a SAS token (provided directly or derived using an account key) or it's based on OAuth2 (which captures the remaining cases).
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 8
- [ ] JDK 11
- [ ] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6765: NIFI-9972 Put blob can now pull from another blob source
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #6765:
URL: https://github.com/apache/nifi/pull/6765#issuecomment-1424292043
Thanks for the reminder @malthe. With focus on the NiFi 1.20 release, I was waiting for some additional input from @turcsanyip.
Considering the proposed changes from another perspective, I'm also wondering about the copy-from-url feature from a usability perspective. Although the end result is putting a blob in the destination, the copy effectively ignores the input FlowFile content if I am following the implementation. For a new user, this may not be the most intuitive approach if the desired capability is to copy from one URL to another. Although the destination URL might be the same in the case of using FlowFile content, the expected flow design would be different. It can be challenging to decide when it makes sense to bundle functionality in one Processors, versus creating a new one, but at this point, a new Processor still seems like a better approach. Do you have any additional thoughts along these lines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6765: NIFI-9972 Put blob can now pull from another blob source
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #6765:
URL: https://github.com/apache/nifi/pull/6765#issuecomment-1404108984
Thanks for the reply @malthe. Although there is some value in co-locating functionality given the same REST endpoint, from a usability and maintenance perspective, a separate Processor might be better. On the other hand, I can see where keeping the implementation in this Processor makes some things easier. The additional authentication handling code for the source warrants some further review.
As @turcsanyip has some experience with these Azure Blob Processors, perhaps he has some additional thoughts about the best approach to support this capability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] malthe commented on pull request #6765: NIFI-9972 Put blob can now pull from another blob source
Posted by "malthe (via GitHub)" <gi...@apache.org>.
malthe commented on PR #6765:
URL: https://github.com/apache/nifi/pull/6765#issuecomment-1423775060
@exceptionfactory bump?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory closed pull request #6765: NIFI-9972 Put blob can now pull from another blob source
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory closed pull request #6765: NIFI-9972 Put blob can now pull from another blob source
URL: https://github.com/apache/nifi/pull/6765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on pull request #6765: NIFI-9972 Put blob can now pull from another blob source
Posted by "turcsanyip (via GitHub)" <gi...@apache.org>.
turcsanyip commented on PR #6765:
URL: https://github.com/apache/nifi/pull/6765#issuecomment-1424901739
@malthe Sorry for my late response here.
I agree with @exceptionfactory that this "copy from url" functionality should rather go into its own processor.
The general contract of 'Put' processors is to upload the FlowFile content into the target external system. To keep the PutBlob processor aligned with this convention, I would not mix extra functionalities with the regular 'Put' role. This way also the configuration would be more straightforward for the users, I believe.
I understand that the same Azure endpoint is used and there are some common code parts but I think it could be handled via moving that code to abstract/util classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6765: NIFI-9972 Put blob can now pull from another blob source
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #6765:
URL: https://github.com/apache/nifi/pull/6765#issuecomment-1424918393
Thanks for the additional perspective @turcsanyip!
Based on the feedback, I am closing this pull request.
@malthe, if you are interested in working on this feature, a new pull request with a Processor named something along the lines of `CopyAzureBlobStorage` sounds useful. Feel free to discuss any concerns or potential design approaches on the Jira issue. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] malthe commented on pull request #6765: NIFI-9972 Put blob can now pull from another blob source
Posted by "malthe (via GitHub)" <gi...@apache.org>.
malthe commented on PR #6765:
URL: https://github.com/apache/nifi/pull/6765#issuecomment-1404004371
@exceptionfactory I thought about it and the justification for me to have it in the Put Processor is that it's the same REST endpoint that Azure provides. The copy source is implemented using special headers on the `PUT` request.
Of course, that's not to say that we shouldn't have a CopyAzureBlobStorage but I think it's a reasonable justification to be honest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org