You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by pvillard31 <gi...@git.apache.org> on 2018/11/22 17:50:52 UTC
[GitHub] nifi pull request #3182: NIFI-5838 - Improve the schema validation method in...
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/3182
NIFI-5838 - Improve the schema validation method in Kite processors
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-5838
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3182.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3182
----
commit 1628dd452f0322f88c811d3b0563a8df7d21dd40
Author: Pierre Villard <pi...@...>
Date: 2018-11-22T17:50:11Z
NIFI-5838 - Improve the schema validation method in Kite processors
----
---
[GitHub] nifi issue #3182: NIFI-5838 - Improve the schema validation method in Kite p...
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/3182
Thanks @pvillard31 for confirming that. I'm +1 and going to merge this. Please update [Migration Guidance](https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance) to note about this behavioral change. A section title like "Migrating from 1.x.x to 1.9.0 (under development, not released yet, subject to be changed)" would be nice.
---
[GitHub] nifi pull request #3182: NIFI-5838 - Improve the schema validation method in...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3182#discussion_r237407789
--- Diff: nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java ---
@@ -101,38 +100,38 @@ protected static Schema getSchema(String uriOrLiteral, Configuration conf) {
return parseSchema(uriOrLiteral);
}
+ if(uri.getScheme() == null) {
+ throw new SchemaNotFoundException("If the schema is not a JSON string, a scheme must be specified in the URI "
+ + "(ex: dataset:, view:, resource:, file:, hdfs:, etc).");
+ }
+
try {
if ("dataset".equals(uri.getScheme()) || "view".equals(uri.getScheme())) {
return Datasets.load(uri).getDataset().getDescriptor().getSchema();
} else if ("resource".equals(uri.getScheme())) {
- try (InputStream in = Resources.getResource(uri.getSchemeSpecificPart())
- .openStream()) {
+ try (InputStream in = Resources.getResource(uri.getSchemeSpecificPart()).openStream()) {
return parseSchema(uri, in);
}
} else {
// try to open the file
Path schemaPath = new Path(uri);
- FileSystem fs = schemaPath.getFileSystem(conf);
- try (InputStream in = fs.open(schemaPath)) {
+ try (InputStream in = schemaPath.getFileSystem(conf).open(schemaPath)) {
--- End diff --
Done - thanks @ijokarumawak
---
[GitHub] nifi pull request #3182: NIFI-5838 - Improve the schema validation method in...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3182
---
[GitHub] nifi pull request #3182: NIFI-5838 - Improve the schema validation method in...
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3182#discussion_r237317639
--- Diff: nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java ---
@@ -101,38 +100,38 @@ protected static Schema getSchema(String uriOrLiteral, Configuration conf) {
return parseSchema(uriOrLiteral);
}
+ if(uri.getScheme() == null) {
+ throw new SchemaNotFoundException("If the schema is not a JSON string, a scheme must be specified in the URI "
+ + "(ex: dataset:, view:, resource:, file:, hdfs:, etc).");
+ }
+
try {
if ("dataset".equals(uri.getScheme()) || "view".equals(uri.getScheme())) {
return Datasets.load(uri).getDataset().getDescriptor().getSchema();
} else if ("resource".equals(uri.getScheme())) {
- try (InputStream in = Resources.getResource(uri.getSchemeSpecificPart())
- .openStream()) {
+ try (InputStream in = Resources.getResource(uri.getSchemeSpecificPart()).openStream()) {
return parseSchema(uri, in);
}
} else {
// try to open the file
Path schemaPath = new Path(uri);
- FileSystem fs = schemaPath.getFileSystem(conf);
- try (InputStream in = fs.open(schemaPath)) {
+ try (InputStream in = schemaPath.getFileSystem(conf).open(schemaPath)) {
--- End diff --
This statement doesn't make FileSystem.close gets called. Please change it to
```
try (FileSystem fs = schemaPath.getFileSystem(conf); InputStream in = fs.open(schemaPath)) {
```
This way, both in.close() and fs.close() will be called.
---
[GitHub] nifi issue #3182: NIFI-5838 - Improve the schema validation method in Kite p...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3182
Oh that's a good point, just pushed a commit to address that. Not sure I'll be able to test that change to confirm that's enough for the reported issue (at least in the comings days).
---
[GitHub] nifi issue #3182: NIFI-5838 - Improve the schema validation method in Kite p...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3182
Hey @ijokarumawak - just had the opportunity to test this on the cluster where I originally faced the issue. Only properly closing the `FileSystem` object didn't help. It's is necessary to also have the invalidation mechanism to prevent NiFi from being unavailable with the BLOCKED threads.
---
[GitHub] nifi issue #3182: NIFI-5838 - Improve the schema validation method in Kite p...
Posted by ijokarumawak <gi...@git.apache.org>.
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/3182
@pvillard31 I understand the intent of this change and it looks good if we want to invalidate schema URL setting if it doesn't have URI.schema.
Just wondering if we close the `FileSystem fs`, too, the lingering `BLOCKED on org.apache.hadoop.ipc.Client$Connection` threads can be discarded? Current code only closes opened input streams, but doesn't close FileSystem.
---