question regarding input stream from reddit.com

[Shahzad K <sh...@gmail.com>]

Hi

My name is Shahzad Karamat, i am trying to read some tweets from http://stream.pushshift.io/ <http://stream.pushshift.io/> into nifi.
I am using a mac and can read the stream using curl -i 'http://stream.pushshift.io/?subreddit=askreddit'
I can get the stream into my terminal and i have also developed system to read this by using Java code.
The question is:
The strings i read from http://stream.pushshift.io/ <http://stream.pushshift.io/>  by using java, how can i make flowFile of this stream to transfer it to a certain relation?

Regards

Shahzad K

Re: question regarding input stream from reddit.com

[Aldrin Piri <al...@gmail.com>]

Shahzad,

Joe is correct in that we do not have anything that maps directly to this
data stream source.

As a means of getting the data into a NiFi flow, you could also consider
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteProcess/index.html
processor.  I have attached a template that uses an ExecuteProcess instance
to perform the functionality you are interested in.

>From the template descripton:

This template makes use of ExecuteProcess to invoke curl and perform
batching on the streaming response provided by pushshift.io.  These are
batched into 1s intervals. This is inexact, and as a result, some results
may get truncated depending on time boundaries.  We perform a very naive
RouteOnContent to filter out those events without the data payload.


This gives a nice proof of concept of how you could interact with the data
source before diving into a custom processor.  As Joe mentioned, a custom
processor might be nice to handle the data format to be cognizant of event
boundaries and would potentially obviate the need for the included
SplitContent processor.  The attached template will discard some events
that fall on those time batch boundaries.

Let us know if you have any more questions or if you are seeing a number of
"similar" data APIs that could potentially be generically supported from a
project standpoint.

--aldrin

On Mon, Jan 11, 2016 at 11:14 AM, Joe Percivall <
joepercivall@yahoo.com.invalid> wrote:

> Hello Shahzad,
>
> Unfortunately the "stream" functionality of pushshift.io doesn't fit into
> any current NiFi processor. Processors work by having an "OnTrigger" method
> that is used to create FlowFiles with each call. This works nicely for
> aspects of the pushshit.io api like "
> https://api.pushshift.io/reddit/search?q=Einstein&limit=100" where it
> returns a single "unit" of information with each http request. If you are
> able to get the same information you need using the base "api" call for
> pushshift instead of "stream that would work best.
>
> Else you may be able to create a custom processor around your java code
> although it may be pretty difficult. You would need to translate the stream
> into chunks of information that would be put into the contents of FlowFiles
> and routed to a relationship using Session.Transter. For more information
> on creating a custom processor check out the developer guide:
> https://nifi.apache.org/developer-guide.html.
>
> Do either of those help or is a general processor that streams over HTTP
> necessary?
>
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joepercivall@yahoo.com
>
>
>
> On Friday, January 8, 2016 11:16 AM, Shahzad K <sh...@gmail.com> wrote:
>
>
>
> Hi
>
> My name is Shahzad Karamat, i am trying to read some tweets from
> http://stream.pushshift.io/ <http://stream.pushshift.io/> into nifi.
> I am using a mac and can read the stream using curl -i '
> http://stream.pushshift.io/?subreddit=askreddit'
> I can get the stream into my terminal and i have also developed system to
> read this by using Java code.
> The question is:
> The strings i read from http://stream.pushshift.io/ <
> http://stream.pushshift.io/>  by using java, how can i make flowFile of
> this stream to transfer it to a certain relation?
>
> Regards
>
> Shahzad K
>

Re: question regarding input stream from reddit.com

[Joe Percivall <jo...@yahoo.com.INVALID>]

Hello Shahzad,

Unfortunately the "stream" functionality of pushshift.io doesn't fit into any current NiFi processor. Processors work by having an "OnTrigger" method that is used to create FlowFiles with each call. This works nicely for aspects of the pushshit.io api like "https://api.pushshift.io/reddit/search?q=Einstein&limit=100" where it returns a single "unit" of information with each http request. If you are able to get the same information you need using the base "api" call for pushshift instead of "stream that would work best.

Else you may be able to create a custom processor around your java code although it may be pretty difficult. You would need to translate the stream into chunks of information that would be put into the contents of FlowFiles and routed to a relationship using Session.Transter. For more information on creating a custom processor check out the developer guide: https://nifi.apache.org/developer-guide.html.

Do either of those help or is a general processor that streams over HTTP necessary?
 
Joe
- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com



On Friday, January 8, 2016 11:16 AM, Shahzad K <sh...@gmail.com> wrote:



Hi

My name is Shahzad Karamat, i am trying to read some tweets from http://stream.pushshift.io/ <http://stream.pushshift.io/> into nifi.
I am using a mac and can read the stream using curl -i 'http://stream.pushshift.io/?subreddit=askreddit'
I can get the stream into my terminal and i have also developed system to read this by using Java code.
The question is:
The strings i read from http://stream.pushshift.io/ <http://stream.pushshift.io/>  by using java, how can i make flowFile of this stream to transfer it to a certain relation?

Regards

Shahzad K