You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "markobean (via GitHub)" <gi...@apache.org> on 2025/01/09 22:53:21 UTC
[PR] NIFI-14141 Accept Expression Language statement in GenerateFlowFile, … [nifi]
markobean opened a new pull request, #9620:
URL: https://github.com/apache/nifi/pull/9620
…File Size property allowing FlowFiles of variable sizes to be generated. Included some examples of EL statements in Additional Details for GenerateFlowFile.
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-14141](https://issues.apache.org/jira/browse/NIFI-14141)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [X] Pull Request based on current revision of the `main` branch
- [X] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [X] Build completed using `mvn clean install -P contrib-check`
- [X] JDK 21
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [X] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14141 Accept Expression Language statement in GenerateFlowFile, … [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9620:
URL: https://github.com/apache/nifi/pull/9620#discussion_r1909749877
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateFlowFile.java:
##########
@@ -73,4 +78,30 @@ public void testDynamicPropertiesToAttributes() {
generatedFlowFile.assertAttributeEquals("mime.type", "application/text");
}
+ @Test
+ public void testExpressionLanguageSupport() {
+ TestRunner runner = TestRunners.newTestRunner(new GenerateFlowFile());
+ runner.setProperty(GenerateFlowFile.FILE_SIZE, "${random():mod(10):plus(1)}B"); // represents a range of 1-10 bytes, inclusive
Review Comment:
Instead of building the unit test against the complexities around the random function, it would be better to use a simple deterministic expression, avoiding the need for multiple runs.
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateFlowFile.java:
##########
@@ -76,7 +76,8 @@ public class GenerateFlowFile extends AbstractProcessor {
.description("The size of the file that will be used")
.required(true)
.defaultValue("0B")
- .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review Comment:
The `DATA_SIZE_VALIDATOR` includes support for expression language, so is this change necessary?
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateFlowFile.java:
##########
@@ -73,4 +78,30 @@ public void testDynamicPropertiesToAttributes() {
generatedFlowFile.assertAttributeEquals("mime.type", "application/text");
}
+ @Test
+ public void testExpressionLanguageSupport() {
+ TestRunner runner = TestRunners.newTestRunner(new GenerateFlowFile());
+ runner.setProperty(GenerateFlowFile.FILE_SIZE, "${random():mod(10):plus(1)}B"); // represents a range of 1-10 bytes, inclusive
+ runner.setProperty(GenerateFlowFile.UNIQUE_FLOWFILES, "true");
+ runner.assertValid();
+
+ // Execute multiple times to ensure adequate distribution of file sizes
+ runner.run(1000);
+
+ runner.assertTransferCount(GenerateFlowFile.SUCCESS, 1000);
+ List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(GenerateFlowFile.SUCCESS);
+ Map<Long, Integer> fileSizeDistribution = new HashMap<>();
+ flowFiles.forEach(ff -> {
+ long fileSize = ff.getSize();
+ fileSizeDistribution.put(fileSize, fileSizeDistribution.getOrDefault(fileSize, 0) + 1);
+ });
+ long minSize = fileSizeDistribution.keySet().stream().min(Long::compareTo).orElse(-1L);
+ long maxSize = fileSizeDistribution.keySet().stream().max(Long::compareTo).orElse(-1L);
+
+ // Assert all file sizes fall within the range and that there exists more than one unique file size value
+ Assertions.assertTrue(minSize > 0 && minSize < maxSize);
+ Assertions.assertTrue(maxSize <= 10);
Review Comment:
The import static should be used for `assertTrue`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14141 Accept Expression Language statement in GenerateFlowFile, … [nifi]
Posted by "markobean (via GitHub)" <gi...@apache.org>.
markobean commented on code in PR #9620:
URL: https://github.com/apache/nifi/pull/9620#discussion_r1909807885
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateFlowFile.java:
##########
@@ -76,7 +76,8 @@ public class GenerateFlowFile extends AbstractProcessor {
.description("The size of the file that will be used")
.required(true)
.defaultValue("0B")
- .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review Comment:
Without that validator within the property, the processor was invalid indicating:
"'File Size' validated against '${random():mod(101)}B' is invalid because 'File Size' is not a supported property or has no Validator associated with it"
As the property is in fact required, this is a reasonable validator to add.
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateFlowFile.java:
##########
@@ -73,4 +78,30 @@ public void testDynamicPropertiesToAttributes() {
generatedFlowFile.assertAttributeEquals("mime.type", "application/text");
}
+ @Test
+ public void testExpressionLanguageSupport() {
+ TestRunner runner = TestRunners.newTestRunner(new GenerateFlowFile());
+ runner.setProperty(GenerateFlowFile.FILE_SIZE, "${random():mod(10):plus(1)}B"); // represents a range of 1-10 bytes, inclusive
Review Comment:
Modified to use `nextInt()` function. But still made 2 batch executions to confirm file size uniqueness (or not) based on `UNIQUE_FLOWFILES` setting. This is slightly out of scope, but the batch processing does not have any other unit test. 2 birds.
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestGenerateFlowFile.java:
##########
@@ -73,4 +78,30 @@ public void testDynamicPropertiesToAttributes() {
generatedFlowFile.assertAttributeEquals("mime.type", "application/text");
}
+ @Test
+ public void testExpressionLanguageSupport() {
+ TestRunner runner = TestRunners.newTestRunner(new GenerateFlowFile());
+ runner.setProperty(GenerateFlowFile.FILE_SIZE, "${random():mod(10):plus(1)}B"); // represents a range of 1-10 bytes, inclusive
+ runner.setProperty(GenerateFlowFile.UNIQUE_FLOWFILES, "true");
+ runner.assertValid();
+
+ // Execute multiple times to ensure adequate distribution of file sizes
+ runner.run(1000);
+
+ runner.assertTransferCount(GenerateFlowFile.SUCCESS, 1000);
+ List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(GenerateFlowFile.SUCCESS);
+ Map<Long, Integer> fileSizeDistribution = new HashMap<>();
+ flowFiles.forEach(ff -> {
+ long fileSize = ff.getSize();
+ fileSizeDistribution.put(fileSize, fileSizeDistribution.getOrDefault(fileSize, 0) + 1);
+ });
+ long minSize = fileSizeDistribution.keySet().stream().min(Long::compareTo).orElse(-1L);
+ long maxSize = fileSizeDistribution.keySet().stream().max(Long::compareTo).orElse(-1L);
+
+ // Assert all file sizes fall within the range and that there exists more than one unique file size value
+ Assertions.assertTrue(minSize > 0 && minSize < maxSize);
+ Assertions.assertTrue(maxSize <= 10);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14141 Accept Expression Language statement in GenerateFlowFile, … [nifi]
Posted by "markobean (via GitHub)" <gi...@apache.org>.
markobean commented on code in PR #9620:
URL: https://github.com/apache/nifi/pull/9620#discussion_r1911264405
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateFlowFile.java:
##########
@@ -187,11 +188,13 @@ protected Collection<ValidationResult> customValidate(ValidationContext validati
+ "text and Unique FlowFiles must be false.").build());
}
+ results.add(StandardValidators.DATA_SIZE_VALIDATOR.validate("File Size", validationContext.getProperty(FILE_SIZE).evaluateAttributeExpressions().getValue(), validationContext));
Review Comment:
Removed.
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateFlowFile.java:
##########
@@ -76,7 +76,8 @@ public class GenerateFlowFile extends AbstractProcessor {
.description("The size of the file that will be used")
.required(true)
.defaultValue("0B")
- .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review Comment:
Thanks for pointing this out. I missed the fact that `DATA_SIZE_VALIDATOR` supports EL. The error I saw was only when no validator was provided at all.
I reverted back to `DATA_SIZE_VALIDATOR` and tested in a running NiFi. All good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14141 Accept Expression Language statement in GenerateFlowFile, … [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #9620:
URL: https://github.com/apache/nifi/pull/9620#discussion_r1910945741
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateFlowFile.java:
##########
@@ -76,7 +76,8 @@ public class GenerateFlowFile extends AbstractProcessor {
.description("The size of the file that will be used")
.required(true)
.defaultValue("0B")
- .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
+ .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Review Comment:
@markobean Reverting this change and running the unit test worked as expected. See `StandardValidators` line 370 where the validator handles the presence of expression language. Do you still see the validation error?
##########
nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateFlowFile.java:
##########
@@ -187,11 +188,13 @@ protected Collection<ValidationResult> customValidate(ValidationContext validati
+ "text and Unique FlowFiles must be false.").build());
}
+ results.add(StandardValidators.DATA_SIZE_VALIDATOR.validate("File Size", validationContext.getProperty(FILE_SIZE).evaluateAttributeExpressions().getValue(), validationContext));
Review Comment:
This line should also be removed after reverting the validator change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-14141 Accept Expression Language statement in GenerateFlowFile, … [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory merged PR #9620:
URL: https://github.com/apache/nifi/pull/9620
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org