You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Utsuk Prani <ut...@gmail.com> on 2020/03/02 12:27:55 UTC
GetTwitter Processor - Need Help
Hi -
I am very new to NiFi. downloaded the latest one - 1.11.3.
For testing I am simply trying to read some tweets and then dump them into
a file.
However from the error I see below, it looks like this might be happening
due to proxy issues as I am behind the company firewall.
I tried to give proxy host inside the 'nifi.properties', however it still
does not work.
Below is how my properties file look like. do I need to make any other
config changes to make it work from behind the firewall ??
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.http.network.interface.default=
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=http://myproxy.domain.com
2020-03-02 17:10:52,862 INFO [Timer-Driven Process Thread-8]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] to run with 1 threads
2020-03-02 17:10:52,927 INFO [Timer-Driven Process Thread-9]
com.twitter.hbc.httpclient.BasicClient New connection executed:
GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b], endpoint:
/1.1/statuses/filter.json?delimited=length&stall_warnings=true
2020-03-02 17:10:52,950 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@731972 // Another save pending =
false
2020-03-02 17:10:53,047 INFO [hosebird-client-io-thread-0]
com.twitter.hbc.httpclient.ClientBase
GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Establishing a
connection
2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
com.twitter.hbc.httpclient.ClientBase
GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Unknown host -
stream.twitter.com
2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
com.twitter.hbc.httpclient.ClientBase
GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] failed to establish
connection properly
2020-03-02 17:10:53,312 INFO [hosebird-client-io-thread-0]
com.twitter.hbc.httpclient.ClientBase
GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Done processing,
preparing to close connection
Re: GetTwitter Processor - Need Help
Posted by Mike Thomsen <mi...@gmail.com>.
Yeah, there's no global proxy configuration that can be applied to all
processors and controller services that use HTTP in some way.
On Tue, Mar 3, 2020 at 1:14 PM Utsuk Prani <ut...@gmail.com> wrote:
> Hi Mike, thanks for your response. It does work when I connect from my
> home and when I am not behind a proxy.
> But I was hoping that the proxy setting should be supported as it is
> normally in most of the web frameworks / apps.
>
> On Mon, Mar 2, 2020 at 10:34 PM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> The properties you referenced there are for accessing the NiFi web
>> interface. Based on what the docs show, it doesn't look like you have the
>> ability to configure a proxy. What I'd recommend is reaching out to your IT
>> department and see what they can do to provide you with the ability to
>> connect directly. They might be able to give you a reverse proxy that lets
>> you securely hit the Twitter endpoints without doing an end run around
>> their security settings or something like that. Alternatively, you could
>> just put your laptop on a hotspot or something like that if you have
>> permission to test.
>>
>> Thanks,
>>
>> Mike
>>
>> On Mon, Mar 2, 2020 at 7:28 AM Utsuk Prani <ut...@gmail.com> wrote:
>>
>>> Hi -
>>>
>>> I am very new to NiFi. downloaded the latest one - 1.11.3.
>>> For testing I am simply trying to read some tweets and then dump them
>>> into a file.
>>>
>>> However from the error I see below, it looks like this might be
>>> happening due to proxy issues as I am behind the company firewall.
>>>
>>> I tried to give proxy host inside the 'nifi.properties', however it
>>> still does not work.
>>> Below is how my properties file look like. do I need to make any other
>>> config changes to make it work from behind the firewall ??
>>>
>>>
>>> # web properties #
>>> nifi.web.war.directory=./lib
>>> nifi.web.http.host=
>>> nifi.web.http.port=8080
>>> nifi.web.http.network.interface.default=
>>> nifi.web.https.host=
>>> nifi.web.https.port=
>>> nifi.web.https.network.interface.default=
>>> nifi.web.jetty.working.directory=./work/jetty
>>> nifi.web.jetty.threads=200
>>> nifi.web.max.header.size=16 KB
>>> nifi.web.proxy.context.path=
>>> nifi.web.proxy.host=http://myproxy.domain.com
>>>
>>>
>>>
>>> 2020-03-02 17:10:52,862 INFO [Timer-Driven Process Thread-8]
>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
>>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] to run with 1 threads
>>> 2020-03-02 17:10:52,927 INFO [Timer-Driven Process Thread-9]
>>> com.twitter.hbc.httpclient.BasicClient New connection executed:
>>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b], endpoint:
>>> /1.1/statuses/filter.json?delimited=length&stall_warnings=true
>>> 2020-03-02 17:10:52,950 INFO [Flow Service Tasks Thread-2]
>>> o.a.nifi.controller.StandardFlowService Saved flow controller
>>> org.apache.nifi.controller.FlowController@731972 // Another save
>>> pending = false
>>> 2020-03-02 17:10:53,047 INFO [hosebird-client-io-thread-0]
>>> com.twitter.hbc.httpclient.ClientBase
>>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Establishing a
>>> connection
>>> 2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
>>> com.twitter.hbc.httpclient.ClientBase
>>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Unknown host -
>>> stream.twitter.com
>>> 2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
>>> com.twitter.hbc.httpclient.ClientBase
>>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] failed to establish
>>> connection properly
>>> 2020-03-02 17:10:53,312 INFO [hosebird-client-io-thread-0]
>>> com.twitter.hbc.httpclient.ClientBase
>>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Done processing,
>>> preparing to close connection
>>>
>>
Re: GetTwitter Processor - Need Help
Posted by Utsuk Prani <ut...@gmail.com>.
Hi Mike, thanks for your response. It does work when I connect from my home
and when I am not behind a proxy.
But I was hoping that the proxy setting should be supported as it is
normally in most of the web frameworks / apps.
On Mon, Mar 2, 2020 at 10:34 PM Mike Thomsen <mi...@gmail.com> wrote:
> The properties you referenced there are for accessing the NiFi web
> interface. Based on what the docs show, it doesn't look like you have the
> ability to configure a proxy. What I'd recommend is reaching out to your IT
> department and see what they can do to provide you with the ability to
> connect directly. They might be able to give you a reverse proxy that lets
> you securely hit the Twitter endpoints without doing an end run around
> their security settings or something like that. Alternatively, you could
> just put your laptop on a hotspot or something like that if you have
> permission to test.
>
> Thanks,
>
> Mike
>
> On Mon, Mar 2, 2020 at 7:28 AM Utsuk Prani <ut...@gmail.com> wrote:
>
>> Hi -
>>
>> I am very new to NiFi. downloaded the latest one - 1.11.3.
>> For testing I am simply trying to read some tweets and then dump them
>> into a file.
>>
>> However from the error I see below, it looks like this might be happening
>> due to proxy issues as I am behind the company firewall.
>>
>> I tried to give proxy host inside the 'nifi.properties', however it still
>> does not work.
>> Below is how my properties file look like. do I need to make any other
>> config changes to make it work from behind the firewall ??
>>
>>
>> # web properties #
>> nifi.web.war.directory=./lib
>> nifi.web.http.host=
>> nifi.web.http.port=8080
>> nifi.web.http.network.interface.default=
>> nifi.web.https.host=
>> nifi.web.https.port=
>> nifi.web.https.network.interface.default=
>> nifi.web.jetty.working.directory=./work/jetty
>> nifi.web.jetty.threads=200
>> nifi.web.max.header.size=16 KB
>> nifi.web.proxy.context.path=
>> nifi.web.proxy.host=http://myproxy.domain.com
>>
>>
>>
>> 2020-03-02 17:10:52,862 INFO [Timer-Driven Process Thread-8]
>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] to run with 1 threads
>> 2020-03-02 17:10:52,927 INFO [Timer-Driven Process Thread-9]
>> com.twitter.hbc.httpclient.BasicClient New connection executed:
>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b], endpoint:
>> /1.1/statuses/filter.json?delimited=length&stall_warnings=true
>> 2020-03-02 17:10:52,950 INFO [Flow Service Tasks Thread-2]
>> o.a.nifi.controller.StandardFlowService Saved flow controller
>> org.apache.nifi.controller.FlowController@731972 // Another save pending
>> = false
>> 2020-03-02 17:10:53,047 INFO [hosebird-client-io-thread-0]
>> com.twitter.hbc.httpclient.ClientBase
>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Establishing a
>> connection
>> 2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
>> com.twitter.hbc.httpclient.ClientBase
>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Unknown host -
>> stream.twitter.com
>> 2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
>> com.twitter.hbc.httpclient.ClientBase
>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] failed to establish
>> connection properly
>> 2020-03-02 17:10:53,312 INFO [hosebird-client-io-thread-0]
>> com.twitter.hbc.httpclient.ClientBase
>> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Done processing,
>> preparing to close connection
>>
>
Re: GetTwitter Processor - Need Help
Posted by Mike Thomsen <mi...@gmail.com>.
The properties you referenced there are for accessing the NiFi web
interface. Based on what the docs show, it doesn't look like you have the
ability to configure a proxy. What I'd recommend is reaching out to your IT
department and see what they can do to provide you with the ability to
connect directly. They might be able to give you a reverse proxy that lets
you securely hit the Twitter endpoints without doing an end run around
their security settings or something like that. Alternatively, you could
just put your laptop on a hotspot or something like that if you have
permission to test.
Thanks,
Mike
On Mon, Mar 2, 2020 at 7:28 AM Utsuk Prani <ut...@gmail.com> wrote:
> Hi -
>
> I am very new to NiFi. downloaded the latest one - 1.11.3.
> For testing I am simply trying to read some tweets and then dump them into
> a file.
>
> However from the error I see below, it looks like this might be happening
> due to proxy issues as I am behind the company firewall.
>
> I tried to give proxy host inside the 'nifi.properties', however it still
> does not work.
> Below is how my properties file look like. do I need to make any other
> config changes to make it work from behind the firewall ??
>
>
> # web properties #
> nifi.web.war.directory=./lib
> nifi.web.http.host=
> nifi.web.http.port=8080
> nifi.web.http.network.interface.default=
> nifi.web.https.host=
> nifi.web.https.port=
> nifi.web.https.network.interface.default=
> nifi.web.jetty.working.directory=./work/jetty
> nifi.web.jetty.threads=200
> nifi.web.max.header.size=16 KB
> nifi.web.proxy.context.path=
> nifi.web.proxy.host=http://myproxy.domain.com
>
>
>
> 2020-03-02 17:10:52,862 INFO [Timer-Driven Process Thread-8]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] to run with 1 threads
> 2020-03-02 17:10:52,927 INFO [Timer-Driven Process Thread-9]
> com.twitter.hbc.httpclient.BasicClient New connection executed:
> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b], endpoint:
> /1.1/statuses/filter.json?delimited=length&stall_warnings=true
> 2020-03-02 17:10:52,950 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@731972 // Another save pending
> = false
> 2020-03-02 17:10:53,047 INFO [hosebird-client-io-thread-0]
> com.twitter.hbc.httpclient.ClientBase
> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Establishing a
> connection
> 2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
> com.twitter.hbc.httpclient.ClientBase
> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Unknown host -
> stream.twitter.com
> 2020-03-02 17:10:53,312 WARN [hosebird-client-io-thread-0]
> com.twitter.hbc.httpclient.ClientBase
> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] failed to establish
> connection properly
> 2020-03-02 17:10:53,312 INFO [hosebird-client-io-thread-0]
> com.twitter.hbc.httpclient.ClientBase
> GetTwitter[id=9b0787de-0170-1000-cc51-b27ec20db86b] Done processing,
> preparing to close connection
>