You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by frett27 <gi...@git.apache.org> on 2017/09/29 05:19:47 UTC

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

GitHub user frett27 opened a pull request:

    https://github.com/apache/nifi/pull/2184

    NIFI-4441 : add maprecord support for avro union types

    Thank you for submitting a contribution to Apache NiFi.
    
    In order to streamline the review of the contribution we ask you
    to ensure the following steps have been taken:
    
    ### For all changes:
    - [ x] Is there a JIRA ticket associated with this PR? Is it referenced 
         in the commit message?
    
    - [x ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
    
    - [x ] Has your PR been rebased against the latest commit within the target branch (typically master)?
    
    - [x ] Is your initial contribution a single, squashed commit?
    
    ### For code changes:
    - [x ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
    - [ x] Have you written or updated unit tests to verify your changes?
    - [ x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? 
    - [ x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
    - [ x] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
    - [ x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
    
    ### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/frett27/nifi nifi-4441

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/2184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2184
    
----
commit d72a1c7d219d1cc12b0e61cb9e0644a6555dcf3c
Author: Patrice Freydiere <fr...@gmail.com>
Date:   2017-09-29T05:08:08Z

    NIFI 4441 : add maprecord support for avro union types

----


---

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/2184#discussion_r142454362
  
    --- Diff: nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java ---
    @@ -691,6 +697,10 @@ private static boolean isCompatibleDataType(final Object value, final DataType d
                         return true;
                     }
                     break;
    +            case MAP:
    +                if (value instanceof MapRecord) {
    --- End diff --
    
    We can probably use the more generic Record instead of MapRecord. We should probably also consider if value instanceof java.util.Map


---

[GitHub] nifi issue #2184: NIFI-4441 : add maprecord support for avro union types

Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the issue:

    https://github.com/apache/nifi/pull/2184
  
    We cannot take a document/source/item from some place unless it has a clear/compatible license for it.  If there isn't one at all or isn't one that is favorable to our needs as an ASF project we cannot use it.  So either we use all originally created source or we cite it.  Does that make sense?


---

[GitHub] nifi issue #2184: NIFI-4441 : add maprecord support for avro union types

Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on the issue:

    https://github.com/apache/nifi/pull/2184
  
    @frett27 thanks for your contribution! It looks like you're provided some test data that is used in a unit test. Any idea what the license would be on this test data? I'm guessing from looking at it that it was provided by osm.org? If licensing information is not available, it may make sense to just create some mock-up data instead?


---

[GitHub] nifi issue #2184: NIFI-4441 : add maprecord support for avro union types

Posted by frett27 <gi...@git.apache.org>.
Github user frett27 commented on the issue:

    https://github.com/apache/nifi/pull/2184
  
    Thank's matt, new pull request associated : #2207, hope this will help the review


---

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

Posted by frett27 <gi...@git.apache.org>.
Github user frett27 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/2184#discussion_r142597985
  
    --- Diff: nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java ---
    @@ -691,6 +697,10 @@ private static boolean isCompatibleDataType(final Object value, final DataType d
                         return true;
                     }
                     break;
    +            case MAP:
    +                if (value instanceof MapRecord) {
    --- End diff --
    
    OK, i'll have a dive on this


---

[GitHub] nifi issue #2184: NIFI-4441 : add maprecord support for avro union types

Posted by frett27 <gi...@git.apache.org>.
Github user frett27 commented on the issue:

    https://github.com/apache/nifi/pull/2184
  
    Hi, markap14, the definition is mine for the tests, and yes on top of the osm definition, i think there is no license associated issue. And yes could be more interesting for have a separate file for ease the reading.



---

[GitHub] nifi issue #2184: NIFI-4441 : add maprecord support for avro union types

Posted by frett27 <gi...@git.apache.org>.
Github user frett27 commented on the issue:

    https://github.com/apache/nifi/pull/2184
  
    Modified the test resource files to remove any projet references, 
    fix style


---

[GitHub] nifi issue #2184: NIFI-4441 : add maprecord support for avro union types

Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the issue:

    https://github.com/apache/nifi/pull/2184
  
    This branch looks out of whack with the current master, can you start with a fresh master and cherry-pick in your commits for this issue? It will help with merging and review, thanks!


---

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

Posted by frett27 <gi...@git.apache.org>.
Github user frett27 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/2184#discussion_r142597913
  
    --- Diff: nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java ---
    @@ -238,5 +242,23 @@ public void testComplicatedRecursiveSchema() {
             // Make sure the 'parent' field has a schema reference back to the original top level record schema
             Assert.assertEquals(recordASchema, ((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
         }
    +    
    +    @Test
    +    public void testMapWithNullSchema() throws IOException {
    +        
    +        Schema recursiveSchema = new Schema.Parser().parse(
    +                "{\"type\":\"record\",\"name\":\"OSMEntity\",\"namespace\":\"org.osm.avro\",\"fields\":[{\"name\":\"osmtype\",\"type\":{\"type\":\"enum\",\"name\":\"OSMType\",\"symbols\":[\"NODE\",\"WAY\",\"POLYGON\",\"RELATION\"]}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"node\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ANode\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"x\",\"type\":\"double\"},{\"name\":\"y\",\"type\":\"double\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"way\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"AComplex\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"geometry\",\"type\":\"bytes\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"polygon\",\"type\
 ":[\"null\",\"AComplex\"]},{\"name\":\"rel\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ARelation\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]},{\"name\":\"related\",\"type\":[\"null\",{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"ARelated\",\"fields\":[{\"name\":\"relatedId\",\"type\":\"long\"},{\"name\":\"type\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"role\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}}]}]}]}]}"
    --- End diff --
    
    OK


---

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/2184#discussion_r142454463
  
    --- Diff: nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/test/java/org/apache/nifi/avro/TestAvroTypeUtil.java ---
    @@ -238,5 +242,23 @@ public void testComplicatedRecursiveSchema() {
             // Make sure the 'parent' field has a schema reference back to the original top level record schema
             Assert.assertEquals(recordASchema, ((RecordDataType)recordBParentField.get().getDataType()).getChildSchema());
         }
    +    
    +    @Test
    +    public void testMapWithNullSchema() throws IOException {
    +        
    +        Schema recursiveSchema = new Schema.Parser().parse(
    +                "{\"type\":\"record\",\"name\":\"OSMEntity\",\"namespace\":\"org.osm.avro\",\"fields\":[{\"name\":\"osmtype\",\"type\":{\"type\":\"enum\",\"name\":\"OSMType\",\"symbols\":[\"NODE\",\"WAY\",\"POLYGON\",\"RELATION\"]}},{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"node\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ANode\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"x\",\"type\":\"double\"},{\"name\":\"y\",\"type\":\"double\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"way\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"AComplex\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"geometry\",\"type\":\"bytes\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]}]}]},{\"name\":\"polygon\",\"type\
 ":[\"null\",\"AComplex\"]},{\"name\":\"rel\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"ARelation\",\"fields\":[{\"name\":\"id\",\"type\":\"long\"},{\"name\":\"fields\",\"type\":[\"null\",{\"type\":\"map\",\"values\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"avro.java.string\":\"String\"}]},{\"name\":\"related\",\"type\":[\"null\",{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"ARelated\",\"fields\":[{\"name\":\"relatedId\",\"type\":\"long\"},{\"name\":\"type\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"role\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}}]}]}]}]}"
    --- End diff --
    
    I think we'd be better off to externalize this into a .avsc file under src/test/resources


---

[GitHub] nifi pull request #2184: NIFI-4441 : add maprecord support for avro union ty...

Posted by frett27 <gi...@git.apache.org>.
Github user frett27 closed the pull request at:

    https://github.com/apache/nifi/pull/2184


---