You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by joemeszaros <gi...@git.apache.org> on 2015/09/10 14:03:33 UTC
[GitHub] nifi pull request: NIFI 945: New property (wrap as array) in avro2...
GitHub user joemeszaros opened a pull request:
https://github.com/apache/nifi/pull/88
NIFI 945: New property (wrap as array) in avro2json converter
Create a new property (wrap as array) in ConvertAvroToJson, which determines how stream of records is exposed: either as a sequence of single Objects (false), writing every Object to a new line, or as an array of Objects. Default value is true, meaning that the Avro content is exposed as an array of Object entries. False value is useful, when you want to write your records as single instances to a target component (e.g. Kafka).
Let's assume you have an Avro content as stream of tracking events ```({"id":"user1", "item":"itemX", "action": "buy"}, {"id":"user2", "item":"itemY", "action": "like"})...{"id":"userN", ...})```.
If wrap as array is false, the converter will expose the records as sequence of single JSON objects:
```
{"id":"user1", "item":"itemX", "action": "buy"}
{"id":"user2", "item":"itemY", "action": "like"}
...
{"id":"userN", ...}
```
Please bear in mind, that the final output is not a valid JSON content. You can then forward this content e.g. to Kafka, where every record will be a single Kafka message.
If wrap as array is true, the output looks like this:
```
[{"id":"user1", "item":"itemX", "action": "buy"},{"id":"user2", "item":"itemY", "action": "like"},...,{"id":"userN", ...}]
```
It is useful when you want to convert your Avro content to a valid JSON array.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ImpressTV/nifi NIFI-945
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/88.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #88
----
commit 75dce9c1ed1018cc14562478fd83d727501700d3
Author: Joe <jo...@impresstv.com>
Date: 2015-09-09T08:40:15Z
New property (wrap as array) in avro2json converter
Create a new property (wrap as array) in ConvertAvroToJson, which determines how stream of records is exposed: either as a sequence of single Objects (false), writing every Object to a new line, or as an array of Objects. Default value is true, meaning that the Avro content is exposed as a sequence of root-level Object entries. False value is useful, when you want to write your records as single intances to a target component (e.g. Kafka).
Let's assume you have an Avro content as stream of records (record1, record2, ...). If wrap as array is false, the converter will expose the records as sequence of single JSON objects:
record1
record2
...
recordN
Please bear in mind, that the final output is not a valid JSON content. You can then forward this content e.g. to Kafka, where every record will be a single Kafka message.
If wrap as array is true, the output looks like this:
[record1,record2,...,recordN]
It is useful when you want to convert your Avro content to a valid JSON array.
commit 96d6c1cc13e15c2467eb0216668eb5e18a7d48af
Author: Joe <jo...@impresstv.com>
Date: 2015-09-10T11:53:25Z
NIFI 945: Test new property in avro2json converter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI 945: New property (wrap as array) in avro2...
Posted by joemeszaros <gi...@git.apache.org>.
Github user joemeszaros commented on the pull request:
https://github.com/apache/nifi/pull/88#issuecomment-150033007
I can confirm the amendments, please find my inline comments at the referenced commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI 945: New property (wrap as array) in avro2...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/88
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI 945: New property (wrap as array) in avro2...
Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on the pull request:
https://github.com/apache/nifi/pull/88#issuecomment-149082304
I commented on https://issues.apache.org/jira/browse/NIFI-945 - looks good, modulo some minor tweaks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI 945: New property (wrap as array) in avro2...
Posted by joemeszaros <gi...@git.apache.org>.
Github user joemeszaros commented on the pull request:
https://github.com/apache/nifi/pull/88#issuecomment-150132380
Thanks for your help!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI 945: New property (wrap as array) in avro2...
Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on the pull request:
https://github.com/apache/nifi/pull/88#issuecomment-149072290
I'm having a heck of a time getting this to merge clean due to whitespace issues (^M newlines)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI 945: New property (wrap as array) in avro2...
Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on the pull request:
https://github.com/apache/nifi/pull/88#issuecomment-149861434
@joemeszaros - I amended your changes to get it to merge clean and for some consistency with other processors in that thread (see commit d1acd152). I just noticed a the getBytes on 140 needs a charset though. Hoping you can look it over and confirm the amendments are representative of your intent?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---