You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "lordgamez (via GitHub)" <gi...@apache.org> on 2025/03/24 12:22:34 UTC
[PR] MINIFICPP-2543 Make S3 operations FIPS compliant [nifi-minifi-cpp]
lordgamez opened a new pull request, #1947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1947
- Update AWS SDK
- Add new Storage Class and Server Side Encryption options
- Add Checksum Algorithm property to PutS3Object
https://issues.apache.org/jira/browse/MINIFICPP-2543
----------------------------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2543 Make S3 operations FIPS compliant [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1947#discussion_r2010270718
##########
PROCESSORS.md:
##########
@@ -2449,6 +2449,7 @@ In the list below, the names of required properties appear in bold. Any other pr
| **Multipart Part Size** | 5 GB | | Specifies the part size for use when the PutS3Multipart Upload API is used. Flow files will be broken into chunks of this size for the upload process, but the last part sent can be smaller since it is not padded. The valid range is 5MB to 5GB. |
| **Multipart Upload AgeOff Interval** | 60 min | | Specifies the interval at which existing multipart uploads in AWS S3 will be evaluated for ageoff. When processor is triggered it will initiate the ageoff evaluation if this interval has been exceeded. |
| **Multipart Upload Max Age Threshold** | 7 days | | Specifies the maximum age for existing multipart uploads in AWS S3. When the ageoff process occurs, any upload older than this threshold will be aborted. |
+| **Checksum Algorithm** | CRC64NVME | CRC32<br/>CRC32C<br/>SHA1<br/>SHA256<br/>CRC64NVME | Checksum algorithm used to verify the uploaded object. |
Review Comment:
Yes, with the latest version with FIPS enabled no additional configuration is needed even for the S3 processors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2543 Make S3 operations FIPS compliant [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1947#discussion_r2010245182
##########
PROCESSORS.md:
##########
@@ -2449,6 +2449,7 @@ In the list below, the names of required properties appear in bold. Any other pr
| **Multipart Part Size** | 5 GB | | Specifies the part size for use when the PutS3Multipart Upload API is used. Flow files will be broken into chunks of this size for the upload process, but the last part sent can be smaller since it is not padded. The valid range is 5MB to 5GB. |
| **Multipart Upload AgeOff Interval** | 60 min | | Specifies the interval at which existing multipart uploads in AWS S3 will be evaluated for ageoff. When processor is triggered it will initiate the ageoff evaluation if this interval has been exceeded. |
| **Multipart Upload Max Age Threshold** | 7 days | | Specifies the maximum age for existing multipart uploads in AWS S3. When the ageoff process occurs, any upload older than this threshold will be aborted. |
+| **Checksum Algorithm** | CRC64NVME | CRC32<br/>CRC32C<br/>SHA1<br/>SHA256<br/>CRC64NVME | Checksum algorithm used to verify the uploaded object. |
Review Comment:
Now that this is added and the default is not MD5, can a user make minifi FIPS compliant by just enabling FIPS support and following the FIPS steps (config generation) in CONFIGURE.md?
##########
extensions/aws/s3/S3Wrapper.h:
##########
@@ -90,6 +100,14 @@ inline constexpr std::array<std::pair<std::string_view, Aws::S3::Model::ObjectCa
{"AwsExecRead", Aws::S3::Model::ObjectCannedACL::aws_exec_read},
}};
+inline constexpr std::array<std::pair<std::string_view, Aws::S3::Model::ChecksumAlgorithm>, 5> CHECKSUM_ALGORITHM_MAP {{
+ {"CRC32", Aws::S3::Model::ChecksumAlgorithm::CRC32},
+ {"CRC32C", Aws::S3::Model::ChecksumAlgorithm::CRC32C},
+ {"SHA1", Aws::S3::Model::ChecksumAlgorithm::SHA1},
+ {"SHA256", Aws::S3::Model::ChecksumAlgorithm::SHA256},
+ {"CRC64NVME", Aws::S3::Model::ChecksumAlgorithm::CRC64NVME}
+}};
Review Comment:
Should we add MD5 here, since it was the default, or would it add no value at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2543 Make S3 operations FIPS compliant [nifi-minifi-cpp]
Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink closed pull request #1947: MINIFICPP-2543 Make S3 operations FIPS compliant
URL: https://github.com/apache/nifi-minifi-cpp/pull/1947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2543 Make S3 operations FIPS compliant [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1947:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1947#discussion_r2010267824
##########
extensions/aws/s3/S3Wrapper.h:
##########
@@ -90,6 +100,14 @@ inline constexpr std::array<std::pair<std::string_view, Aws::S3::Model::ObjectCa
{"AwsExecRead", Aws::S3::Model::ObjectCannedACL::aws_exec_read},
}};
+inline constexpr std::array<std::pair<std::string_view, Aws::S3::Model::ChecksumAlgorithm>, 5> CHECKSUM_ALGORITHM_MAP {{
+ {"CRC32", Aws::S3::Model::ChecksumAlgorithm::CRC32},
+ {"CRC32C", Aws::S3::Model::ChecksumAlgorithm::CRC32C},
+ {"SHA1", Aws::S3::Model::ChecksumAlgorithm::SHA1},
+ {"SHA256", Aws::S3::Model::ChecksumAlgorithm::SHA256},
+ {"CRC64NVME", Aws::S3::Model::ChecksumAlgorithm::CRC64NVME}
+}};
Review Comment:
Previously MD5 hash was added in the Content-MD5 header if no Checksum Algorithm was provided, but MD5 was not an option that could be selected for the Checksum Algorithm. With the latest SDK version this is no longer the case, if no Checksum Algorithm is provided then CRC64NVME is selected and used by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org