You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/02/16 18:51:23 UTC
[GitHub] [nifi] pgyori opened a new pull request #4825: NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
pgyori opened a new pull request #4825:
URL: https://github.com/apache/nifi/pull/4825
https://issues.apache.org/jira/browse/NIFI-7969
#### Description of PR
This commit adds Force Types From Schema property to the ValidateRecord processor and contains documentation update for the processor.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-XXXX** where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4825: NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4825:
URL: https://github.com/apache/nifi/pull/4825#discussion_r584702009
##########
File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/docs/org.apache.nifi.processors.standard.ValidateRecord/additionalDetails.html
##########
@@ -29,13 +29,23 @@
<h2>Examples for the effect of Force Types From Schema property</h2>
<p>
- The value of the Force Types From Schema property has an effect on what
+ The processor first reads the data from the incoming FlowFile using the specified Record Reader,
+ which uses a schema. Then, depending on the value of the Schema Access Strategy property, the processor
+ can either use the reader's schema, or a different schema to validate the data against.
+ After that, the processor writes the data into the outgoing FlowFile using the specified
+ Record Writer. If the data is valid, the validation schema is used by the writer.
+ If the data is invalid, the writer uses the reader's schema.
+ The <b>Force Types From Schema</b> property affects the first step:
Review comment:
Thank you! The fix is in my new commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] pgyori commented on a change in pull request #4825: NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
Posted by GitBox <gi...@apache.org>.
pgyori commented on a change in pull request #4825:
URL: https://github.com/apache/nifi/pull/4825#discussion_r582298934
##########
File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateRecord.java
##########
@@ -180,6 +180,19 @@
.defaultValue("true")
.required(true)
.build();
+ static final PropertyDescriptor COERCE_TYPES = new PropertyDescriptor.Builder()
+ .name("coerce-types")
+ .displayName("Force Types From Schema")
+ .description("If enabled, the processor will coerce every field to the type specified in the schema. "
Review comment:
Thank you! I pushed a new commit with the clarification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4825: NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4825:
URL: https://github.com/apache/nifi/pull/4825
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4825: NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4825:
URL: https://github.com/apache/nifi/pull/4825#discussion_r580515586
##########
File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateRecord.java
##########
@@ -180,6 +180,19 @@
.defaultValue("true")
.required(true)
.build();
+ static final PropertyDescriptor COERCE_TYPES = new PropertyDescriptor.Builder()
+ .name("coerce-types")
+ .displayName("Force Types From Schema")
+ .description("If enabled, the processor will coerce every field to the type specified in the schema. "
Review comment:
I would mention in the description (and maybe also in the property name) that the schema here is the Record Reader's schema, not the validation schema (which may be a different one).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4825: NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
Posted by GitBox <gi...@apache.org>.
turcsanyip commented on a change in pull request #4825:
URL: https://github.com/apache/nifi/pull/4825#discussion_r584551485
##########
File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateRecord.java
##########
@@ -172,22 +172,24 @@
static final PropertyDescriptor STRICT_TYPE_CHECKING = new PropertyDescriptor.Builder()
.name("strict-type-checking")
.displayName("Strict Type Checking")
- .description("If the incoming data has a Record where a field is not of the correct type, this property determine whether how to handle the Record. "
- + "If true, the Record will still be considered invalid. If false, the Record will be considered valid and the field will be coerced into the "
- + "correct type (if possible, according to the type coercion supported by the Record Writer).")
+ .description("If the incoming data has a Record where a field is not of the correct type, this property determines how to handle the Record. "
+ + "If true, the Record will be considered invalid. If false, the Record will be considered valid and the field will be coerced into the "
+ + "correct type (if possible, according to the type coercion supported by the Record Writer). "
+ + "This property controls how the data is validated against the validation schema.")
.expressionLanguageSupported(ExpressionLanguageScope.NONE)
.allowableValues("true", "false")
.defaultValue("true")
.required(true)
.build();
static final PropertyDescriptor COERCE_TYPES = new PropertyDescriptor.Builder()
.name("coerce-types")
- .displayName("Force Types From Schema")
- .description("If enabled, the processor will coerce every field to the type specified in the schema. "
- + "If the value of the field cannot be coerced to the type, the field will be skipped and will not "
- + "appear in the output. "
+ .displayName("Force Types From Reader's Schema")
+ .description("If enabled, the processor will coerce every field to the type specified in the Reader's schema. "
+ + "If the value of a field cannot be coerced to the type, the field will be skipped (will not be read from the input data), "
+ + "thus will not appear in the output. "
+ "If not enabled, then every field will appear in the output but their types may differ from what is "
- + "specified in the schema. For details please see the Additional Details page of the processor's Help.")
+ + "specified in the schema. For details please see the Additional Details page of the processor's Help."
+ + "This property controls how the data is read by the specified Record Reader.")
Review comment:
Missing space before the sentence.
##########
File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/resources/docs/org.apache.nifi.processors.standard.ValidateRecord/additionalDetails.html
##########
@@ -29,13 +29,23 @@
<h2>Examples for the effect of Force Types From Schema property</h2>
<p>
- The value of the Force Types From Schema property has an effect on what
+ The processor first reads the data from the incoming FlowFile using the specified Record Reader,
+ which uses a schema. Then, depending on the value of the Schema Access Strategy property, the processor
+ can either use the reader's schema, or a different schema to validate the data against.
+ After that, the processor writes the data into the outgoing FlowFile using the specified
+ Record Writer. If the data is valid, the validation schema is used by the writer.
+ If the data is invalid, the writer uses the reader's schema.
+ The <b>Force Types From Schema</b> property affects the first step:
Review comment:
Property name has changed: `Force Types From Reader's Schema`
It should be refreshed in this doc too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org