You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Pat Trainor <pa...@gmail.com> on 2016/06/18 21:20:18 UTC
Parsing/extracting values from GetTwitter output...
I can't seem to pull values out of a raw Tweet...
Starting here with GetTwitter:
[image: Inline image 1]
The output is like this:
{"created_at":"Sun Jun 12 18:24:10 +0000
2016","id":742059937359106048,"id_str":"742059937359106048","text":"RT
@WillMcAvoyACN: Our friends at @OpenSecretsDC have a complete list of those
who have recieved money from the NRA this election cycle
htt\u2026","source":"\u003ca href=\"http:\/\/twitter.com\/
[...]
I then have an EvaluateJsonPath processor:
[image: Inline image 2]
So I would expect there to be 5 attributes pulled from the raw tweet... as
flowfile attributes?
This output is then parsed/ran through a java program that tries to display
all the parts of the flowfile, and send just the text (message/tweet)
along... With a lot cut out, the code bits of most interest include:
for (ConsumerRecord<String, String> record : records) {
p("--------------------------------------------------------------------------------");
p("[main] in for(), got topic: " + record.topic());
try {
// record.topic and record.value seem the most useful
switch (record.topic()) {
case "fast-messages":
/**
* This is where we run the text through the parser(s):
*/
// flowfile = flowfile.toUpperCase(); // just to see
the change...
JSONObject jsonObj = new JSONObject(record.value());
p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
// System.out.println("ID: " + jsonObj.getString("id"));
flowfile = jsonObj.getString("text");
System.out.println("TEXT: " + jsonObj.getString("text"));
System.out.println("CREATED_AT: " +
jsonObj.getString("created_at"));
// System.out.println("USER Handle: " +
jsonObj.getString("screen_name"));
// System.out.println("User Name: " +
jsonObj.getString("user_name"));
p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
...I can get text from *created_at* and *text*, but no others.
Here is the output for the above (only using 2 attributes):
2016-06-18T14:01:54.137 [main] in for(), got topic: fast-messages
2016-06-18T14:01:54.137
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TEXT: RT @thehill: Poll: Clinton holds double-digit lead over Trump in
general election matchup https://t.co/tQdO9rmWIo https://t.co/VFWxFMIEs5
CREATED_AT: Sun Jun 12 18:24:08 +0000 2016
2016-06-18T14:01:54.137
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
text & created at... But now I'll uncomment *screen_name*:
016-06-18T17:18:03.480 [main] in for(), got topic: fast-messages
2016-06-18T17:18:03.484
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
TEXT: RT @WillMcAvoyACN: Has received $1,000 in direct donations from the
NRA this election cycle. https://t.co/vOS7HcYity
CREATED_AT: Sun Jun 12 18:24:10 +0000 2016
EXCEPTION: org.json.JSONException: JSONObject["screen_name"] not found.
JSON has nested values, but I don't know enough to figure out how to
assign/grab them in a processor...0
...which makes sense, I guess... The problem is, I don't know how to pull
out attributes from the raw tweet, and make them available later...
How can I approach this need?
Thanks
in advance
!
pat <http://about.me/PatTrainor>
( ͡° ͜ʖ ͡°)
Re: Parsing/extracting values from GetTwitter output...
Posted by Yolanda Davis <yo...@gmail.com>.
No worries :)! If anything I hope it's helpful. Let me know if you have
any questions or I'm sure others can chime in as well.
-yolanda
On Sat, Jun 18, 2016 at 7:34 PM, Pat Trainor <pa...@gmail.com> wrote:
> Yolanda,
>
> WOW! A lot to take in... Am doing so now, so I'm writing so you don't
> think I'm ignoring!
>
> pat
> :)
>
>
> Thanks!
>
> pat <http://about.me/PatTrainor>
> ( ͡° ͜ʖ ͡°)
>
> LinkedIn <http://LinkedIn.com/in/PatTrainor>
> Hobby/Fun Blog <http://blog.atcp.us>
> Sales Engineering University <http://seu.atcp.us>
>
>
>
> "A wise man can learn more from a foolish *question *than a fool can
> learn from a wise *answer*". ~ Bruce Lee.
>
> On Sat, Jun 18, 2016 at 7:32 PM, Yolanda Davis <yo...@gmail.com>
> wrote:
>
>> Hi Pat,
>>
>> I created a flow to test out what you are seeing. Not quite sure the java
>> program you are using to parse out the flow file contents but maybe this
>> will help:
>>
>>
>>
>> The flow shown above is based on code in the upcoming version of NiFi
>> however many of these processors are available in 0.7. If you focus on
>> the lower half I used the EvaluateJsonPath along with an AttributesToJson
>> processor, which simply converts flow file attributes I choose into json
>> and places it in flow file content (but there's an option to place it in an
>> attribute as well) . I then save the content as a file in a directory using
>> the PutFile processor so I could take a look at it later. Here are my
>> configs (some based on what you sent):
>>
>>
>> (EvaluateJsonPath - configuration. Note the language there as well, I
>> saw that Twitter had lang instead of language)
>>
>>
>> (AttributesToJson - here I selected the property names that were defined
>> in the EvaluateJsonPath that were saved as attributes)
>>
>> Here is an example of the file saved:
>>
>> {"twitter_handle":"jobfindlybal","twitter.msg":"Telemarketer State Farm
>> Agent Team Member Jobs in Baltimore, MD #Baltimore #MD #jobs #jobsearch
>> https://t.co/dLhSez6fsT
>> ","twitter.id_str":"744306388676059136","language":"en"}
>>
>> One thing I noted in your screenshot above is even though you are
>> defining properties in the EvaluateJsonPath, the java program appears to be
>> looking for the actual values in the original tweet. I would suggest
>> looking for the property names you created to see if that helps. Another
>> suggestion would be to leverage the AttributeToJson Processor right after
>> your EvaluateJsonPath processor so that you can select the attribute values
>> you want to use and pass them along as flowfile content for the java
>> program to consume.
>>
>> Finally I'm not sure what version of NiFi you are using, however in
>> version 0.7 there is a JoltTransformJSON processor that can be used to
>> transform incoming JSON (using JOLT specification) to only what you want.
>> It basically can transform the schema to a paired down format or even a new
>> one. That can probably eliminate some of the processing steps in NiFi as
>> well.
>>
>> I hope this helps! Please let me know if you have any questions.
>>
>> Best Regards,
>>
>> -yolanda
>>
>> On Sat, Jun 18, 2016 at 5:20 PM, Pat Trainor <pa...@gmail.com>
>> wrote:
>>
>>> I can't seem to pull values out of a raw Tweet...
>>>
>>> Starting here with GetTwitter:
>>>
>>> [image: Inline image 1]
>>>
>>> The output is like this:
>>>
>>> {"created_at":"Sun Jun 12 18:24:10 +0000
>>> 2016","id":742059937359106048,"id_str":"742059937359106048","text":"RT
>>> @WillMcAvoyACN: Our friends at @OpenSecretsDC have a complete list of those
>>> who have recieved money from the NRA this election cycle
>>> htt\u2026","source":"\u003ca href=\"http:\/\/twitter.com\/
>>> [...]
>>>
>>> I then have an EvaluateJsonPath processor:
>>>
>>> [image: Inline image 2]
>>>
>>> So I would expect there to be 5 attributes pulled from the raw tweet...
>>> as flowfile attributes?
>>>
>>> This output is then parsed/ran through a java program that tries to
>>> display all the parts of the flowfile, and send just the text
>>> (message/tweet) along... With a lot cut out, the code bits of most interest
>>> include:
>>>
>>> for (ConsumerRecord<String, String> record : records) {
>>> p("--------------------------------------------------------------------------------");
>>> p("[main] in for(), got topic: " + record.topic());
>>> try {
>>> // record.topic and record.value seem the most useful
>>> switch (record.topic()) {
>>> case "fast-messages":
>>> /**
>>> * This is where we run the text through the parser(s):
>>> */
>>> // flowfile = flowfile.toUpperCase(); // just to see the change...
>>>
>>> JSONObject jsonObj = new JSONObject(record.value());
>>>
>>> p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
>>> // System.out.println("ID: " + jsonObj.getString("id"));
>>> flowfile = jsonObj.getString("text");
>>> System.out.println("TEXT: " + jsonObj.getString("text"));
>>> System.out.println("CREATED_AT: " + jsonObj.getString("created_at"));
>>> // System.out.println("USER Handle: " + jsonObj.getString("screen_name"));
>>> // System.out.println("User Name: " + jsonObj.getString("user_name"));
>>> p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
>>>
>>> ...I can get text from *created_at* and *text*, but no others.
>>> Here is the output for the above (only using 2 attributes):
>>>
>>> 2016-06-18T14:01:54.137 [main] in for(), got topic: fast-messages
>>> 2016-06-18T14:01:54.137
>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> TEXT: RT @thehill: Poll: Clinton holds double-digit lead over Trump in
>>> general election matchup https://t.co/tQdO9rmWIo https://t.co/VFWxFMIEs5
>>> CREATED_AT: Sun Jun 12 18:24:08 +0000 2016
>>> 2016-06-18T14:01:54.137
>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>> text & created at... But now I'll uncomment *screen_name*:
>>>
>>> 016-06-18T17:18:03.480 [main] in for(), got topic: fast-messages
>>> 2016-06-18T17:18:03.484
>>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> TEXT: RT @WillMcAvoyACN: Has received $1,000 in direct donations from
>>> the NRA this election cycle. https://t.co/vOS7HcYity
>>> CREATED_AT: Sun Jun 12 18:24:10 +0000 2016
>>> EXCEPTION: org.json.JSONException: JSONObject["screen_name"] not found.
>>>
>>> JSON has nested values, but I don't know enough to figure out how to
>>> assign/grab them in a processor...0
>>>
>>> ...which makes sense, I guess... The problem is, I don't know how to
>>> pull out attributes from the raw tweet, and make them available later...
>>>
>>> How can I approach this need?
>>>
>>> Thanks
>>> in advance
>>> !
>>>
>>> pat <http://about.me/PatTrainor>
>>> ( ͡° ͜ʖ ͡°)
>>>
>>>
>>
>>
>> --
>> --
>> yolanda.m.davis@gmail.com
>> @YolandaMDavis
>>
>>
>
--
--
yolanda.m.davis@gmail.com
@YolandaMDavis
Re: Parsing/extracting values from GetTwitter output...
Posted by Pat Trainor <pa...@gmail.com>.
Yolanda,
WOW! A lot to take in... Am doing so now, so I'm writing so you don't think
I'm ignoring!
pat
:)
Thanks!
pat <http://about.me/PatTrainor>
( ͡° ͜ʖ ͡°)
LinkedIn <http://LinkedIn.com/in/PatTrainor>
Hobby/Fun Blog <http://blog.atcp.us>
Sales Engineering University <http://seu.atcp.us>
"A wise man can learn more from a foolish *question *than a fool can learn
from a wise *answer*". ~ Bruce Lee.
On Sat, Jun 18, 2016 at 7:32 PM, Yolanda Davis <yo...@gmail.com>
wrote:
> Hi Pat,
>
> I created a flow to test out what you are seeing. Not quite sure the java
> program you are using to parse out the flow file contents but maybe this
> will help:
>
>
>
> The flow shown above is based on code in the upcoming version of NiFi
> however many of these processors are available in 0.7. If you focus on
> the lower half I used the EvaluateJsonPath along with an AttributesToJson
> processor, which simply converts flow file attributes I choose into json
> and places it in flow file content (but there's an option to place it in an
> attribute as well) . I then save the content as a file in a directory using
> the PutFile processor so I could take a look at it later. Here are my
> configs (some based on what you sent):
>
>
> (EvaluateJsonPath - configuration. Note the language there as well, I
> saw that Twitter had lang instead of language)
>
>
> (AttributesToJson - here I selected the property names that were defined
> in the EvaluateJsonPath that were saved as attributes)
>
> Here is an example of the file saved:
>
> {"twitter_handle":"jobfindlybal","twitter.msg":"Telemarketer State Farm
> Agent Team Member Jobs in Baltimore, MD #Baltimore #MD #jobs #jobsearch
> https://t.co/dLhSez6fsT
> ","twitter.id_str":"744306388676059136","language":"en"}
>
> One thing I noted in your screenshot above is even though you are defining
> properties in the EvaluateJsonPath, the java program appears to be looking
> for the actual values in the original tweet. I would suggest looking for
> the property names you created to see if that helps. Another suggestion
> would be to leverage the AttributeToJson Processor right after your
> EvaluateJsonPath processor so that you can select the attribute values you
> want to use and pass them along as flowfile content for the java program to
> consume.
>
> Finally I'm not sure what version of NiFi you are using, however in
> version 0.7 there is a JoltTransformJSON processor that can be used to
> transform incoming JSON (using JOLT specification) to only what you want.
> It basically can transform the schema to a paired down format or even a new
> one. That can probably eliminate some of the processing steps in NiFi as
> well.
>
> I hope this helps! Please let me know if you have any questions.
>
> Best Regards,
>
> -yolanda
>
> On Sat, Jun 18, 2016 at 5:20 PM, Pat Trainor <pa...@gmail.com>
> wrote:
>
>> I can't seem to pull values out of a raw Tweet...
>>
>> Starting here with GetTwitter:
>>
>> [image: Inline image 1]
>>
>> The output is like this:
>>
>> {"created_at":"Sun Jun 12 18:24:10 +0000
>> 2016","id":742059937359106048,"id_str":"742059937359106048","text":"RT
>> @WillMcAvoyACN: Our friends at @OpenSecretsDC have a complete list of those
>> who have recieved money from the NRA this election cycle
>> htt\u2026","source":"\u003ca href=\"http:\/\/twitter.com\/
>> [...]
>>
>> I then have an EvaluateJsonPath processor:
>>
>> [image: Inline image 2]
>>
>> So I would expect there to be 5 attributes pulled from the raw tweet...
>> as flowfile attributes?
>>
>> This output is then parsed/ran through a java program that tries to
>> display all the parts of the flowfile, and send just the text
>> (message/tweet) along... With a lot cut out, the code bits of most interest
>> include:
>>
>> for (ConsumerRecord<String, String> record : records) {
>> p("--------------------------------------------------------------------------------");
>> p("[main] in for(), got topic: " + record.topic());
>> try {
>> // record.topic and record.value seem the most useful
>> switch (record.topic()) {
>> case "fast-messages":
>> /**
>> * This is where we run the text through the parser(s):
>> */
>> // flowfile = flowfile.toUpperCase(); // just to see the change...
>>
>> JSONObject jsonObj = new JSONObject(record.value());
>>
>> p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
>> // System.out.println("ID: " + jsonObj.getString("id"));
>> flowfile = jsonObj.getString("text");
>> System.out.println("TEXT: " + jsonObj.getString("text"));
>> System.out.println("CREATED_AT: " + jsonObj.getString("created_at"));
>> // System.out.println("USER Handle: " + jsonObj.getString("screen_name"));
>> // System.out.println("User Name: " + jsonObj.getString("user_name"));
>> p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
>>
>> ...I can get text from *created_at* and *text*, but no others.
>> Here is the output for the above (only using 2 attributes):
>>
>> 2016-06-18T14:01:54.137 [main] in for(), got topic: fast-messages
>> 2016-06-18T14:01:54.137
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> TEXT: RT @thehill: Poll: Clinton holds double-digit lead over Trump in
>> general election matchup https://t.co/tQdO9rmWIo https://t.co/VFWxFMIEs5
>> CREATED_AT: Sun Jun 12 18:24:08 +0000 2016
>> 2016-06-18T14:01:54.137
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> text & created at... But now I'll uncomment *screen_name*:
>>
>> 016-06-18T17:18:03.480 [main] in for(), got topic: fast-messages
>> 2016-06-18T17:18:03.484
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> TEXT: RT @WillMcAvoyACN: Has received $1,000 in direct donations from the
>> NRA this election cycle. https://t.co/vOS7HcYity
>> CREATED_AT: Sun Jun 12 18:24:10 +0000 2016
>> EXCEPTION: org.json.JSONException: JSONObject["screen_name"] not found.
>>
>> JSON has nested values, but I don't know enough to figure out how to
>> assign/grab them in a processor...0
>>
>> ...which makes sense, I guess... The problem is, I don't know how to pull
>> out attributes from the raw tweet, and make them available later...
>>
>> How can I approach this need?
>>
>> Thanks
>> in advance
>> !
>>
>> pat <http://about.me/PatTrainor>
>> ( ͡° ͜ʖ ͡°)
>>
>>
>
>
> --
> --
> yolanda.m.davis@gmail.com
> @YolandaMDavis
>
>
Re: Parsing/extracting values from GetTwitter output...
Posted by Yolanda Davis <yo...@gmail.com>.
Hi Pat,
I created a flow to test out what you are seeing. Not quite sure the java
program you are using to parse out the flow file contents but maybe this
will help:
The flow shown above is based on code in the upcoming version of NiFi
however many of these processors are available in 0.7. If you focus on
the lower half I used the EvaluateJsonPath along with an AttributesToJson
processor, which simply converts flow file attributes I choose into json
and places it in flow file content (but there's an option to place it in an
attribute as well) . I then save the content as a file in a directory using
the PutFile processor so I could take a look at it later. Here are my
configs (some based on what you sent):
(EvaluateJsonPath - configuration. Note the language there as well, I saw
that Twitter had lang instead of language)
(AttributesToJson - here I selected the property names that were defined
in the EvaluateJsonPath that were saved as attributes)
Here is an example of the file saved:
{"twitter_handle":"jobfindlybal","twitter.msg":"Telemarketer State Farm
Agent Team Member Jobs in Baltimore, MD #Baltimore #MD #jobs #jobsearch
https://t.co/dLhSez6fsT
","twitter.id_str":"744306388676059136","language":"en"}
One thing I noted in your screenshot above is even though you are defining
properties in the EvaluateJsonPath, the java program appears to be looking
for the actual values in the original tweet. I would suggest looking for
the property names you created to see if that helps. Another suggestion
would be to leverage the AttributeToJson Processor right after your
EvaluateJsonPath processor so that you can select the attribute values you
want to use and pass them along as flowfile content for the java program to
consume.
Finally I'm not sure what version of NiFi you are using, however in version
0.7 there is a JoltTransformJSON processor that can be used to transform
incoming JSON (using JOLT specification) to only what you want. It
basically can transform the schema to a paired down format or even a new
one. That can probably eliminate some of the processing steps in NiFi as
well.
I hope this helps! Please let me know if you have any questions.
Best Regards,
-yolanda
On Sat, Jun 18, 2016 at 5:20 PM, Pat Trainor <pa...@gmail.com> wrote:
> I can't seem to pull values out of a raw Tweet...
>
> Starting here with GetTwitter:
>
> [image: Inline image 1]
>
> The output is like this:
>
> {"created_at":"Sun Jun 12 18:24:10 +0000
> 2016","id":742059937359106048,"id_str":"742059937359106048","text":"RT
> @WillMcAvoyACN: Our friends at @OpenSecretsDC have a complete list of those
> who have recieved money from the NRA this election cycle
> htt\u2026","source":"\u003ca href=\"http:\/\/twitter.com\/
> [...]
>
> I then have an EvaluateJsonPath processor:
>
> [image: Inline image 2]
>
> So I would expect there to be 5 attributes pulled from the raw tweet... as
> flowfile attributes?
>
> This output is then parsed/ran through a java program that tries to
> display all the parts of the flowfile, and send just the text
> (message/tweet) along... With a lot cut out, the code bits of most interest
> include:
>
> for (ConsumerRecord<String, String> record : records) {
> p("--------------------------------------------------------------------------------");
> p("[main] in for(), got topic: " + record.topic());
> try {
> // record.topic and record.value seem the most useful
> switch (record.topic()) {
> case "fast-messages":
> /**
> * This is where we run the text through the parser(s):
> */
> // flowfile = flowfile.toUpperCase(); // just to see the change...
>
> JSONObject jsonObj = new JSONObject(record.value());
>
> p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
> // System.out.println("ID: " + jsonObj.getString("id"));
> flowfile = jsonObj.getString("text");
> System.out.println("TEXT: " + jsonObj.getString("text"));
> System.out.println("CREATED_AT: " + jsonObj.getString("created_at"));
> // System.out.println("USER Handle: " + jsonObj.getString("screen_name"));
> // System.out.println("User Name: " + jsonObj.getString("user_name"));
> p("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
>
> ...I can get text from *created_at* and *text*, but no others.
> Here is the output for the above (only using 2 attributes):
>
> 2016-06-18T14:01:54.137 [main] in for(), got topic: fast-messages
> 2016-06-18T14:01:54.137
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> TEXT: RT @thehill: Poll: Clinton holds double-digit lead over Trump in
> general election matchup https://t.co/tQdO9rmWIo https://t.co/VFWxFMIEs5
> CREATED_AT: Sun Jun 12 18:24:08 +0000 2016
> 2016-06-18T14:01:54.137
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> text & created at... But now I'll uncomment *screen_name*:
>
> 016-06-18T17:18:03.480 [main] in for(), got topic: fast-messages
> 2016-06-18T17:18:03.484
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> TEXT: RT @WillMcAvoyACN: Has received $1,000 in direct donations from the
> NRA this election cycle. https://t.co/vOS7HcYity
> CREATED_AT: Sun Jun 12 18:24:10 +0000 2016
> EXCEPTION: org.json.JSONException: JSONObject["screen_name"] not found.
>
> JSON has nested values, but I don't know enough to figure out how to
> assign/grab them in a processor...0
>
> ...which makes sense, I guess... The problem is, I don't know how to pull
> out attributes from the raw tweet, and make them available later...
>
> How can I approach this need?
>
> Thanks
> in advance
> !
>
> pat <http://about.me/PatTrainor>
> ( ͡° ͜ʖ ͡°)
>
>
--
--
yolanda.m.davis@gmail.com
@YolandaMDavis