You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Igor Kravzov <ig...@gmail.com> on 2016/05/26 20:12:14 UTC
PutKafka warning - why?
Why I am getting warning bellow?
[image: Inline image 2]
Re: PutKafka warning - why?
Posted by Igor Kravzov <ig...@gmail.com>.
Thanks Oleg. Just want to clarify: PutKafka is unusable in 0.6.1?
On Thu, May 26, 2016 at 4:49 PM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:
> Igor
>
> Thanks a lot
>
> That’s clearly a bug, simply because NPE originates form our code (must
> never happen). Thankfully it was fixed already and will be available in 0.7
> release. But if you brave enough you can build from master and use it in
> your current environment in the meantime (let me know if you need help with
> it).
> Otherwise, the 0.7 is not to far away.
>
> Cheers
> Oleg
>
> On May 26, 2016, at 4:39 PM, Igor Kravzov <ig...@gmail.com> wrote:
>
> Actually I wanted to write new email, but will ask here.
> I am trying to use PutKafka with Hortonworks HDP 2.4. Latest. The template
> is simple and shown bellow. The warning above I received when tried to
> change processor's configuration and start again.
>
> <image.png>
>
> Actually I am unable to send file to Kafka. Getting exception bellow.
>
>
> 2016-05-26 16:25:56,227 INFO [StandardProcessScheduler Thread-4]
> o.a.k.clients.producer.ProducerConfig ProducerConfig values:
> compression.type = none
> metric.reporters = []
> metadata.max.age.ms = 300000
> metadata.fetch.timeout.ms = 30000
> acks = 0
> batch.size = 1
> reconnect.backoff.ms = 10
> bootstrap.servers = [10.1.132.113:6667]
> receive.buffer.bytes = 32768
> retry.backoff.ms = 100
> buffer.memory = 5242880
> timeout.ms = 30000
> key.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> retries = 0
> max.request.size = 1048576
> block.on.buffer.full = true
> value.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> metrics.sample.window.ms = 30000
> send.buffer.bytes = 131072
> max.in.flight.requests.per.connection = 5
> metrics.num.samples = 2
> linger.ms = 5000
> client.id = MyTEst
>
> 2016-05-26 16:25:56,443 INFO [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@6e7a6be5 // Another save
> pending = false
> 2016-05-26 16:26:00,771 WARN [StandardProcessScheduler Thread-4]
> o.a.k.clients.producer.ProducerConfig The configuration partitioner.class =
> null was supplied but isn't a known config.
> 2016-05-26 16:26:00,771 INFO [StandardProcessScheduler Thread-8]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] to run with 1 threads
> 2016-05-26 16:26:04,979 INFO [pool-16-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository
> 2016-05-26 16:26:05,216 INFO [pool-16-thread-1]
> org.wali.MinimalLockingWriteAheadLog
> org.wali.MinimalLockingWriteAheadLog@1227013b checkpointed with 1 Records
> and 0 Swap Files in 237 milliseconds (Stop-the-world time = 110
> milliseconds, Clear Edit Logs time = 124 millis), max Transaction ID
> 252557730
> 2016-05-26 16:26:05,216 INFO [pool-16-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
> Repository with 1 records in 237 milliseconds
> 2016-05-26 16:27:00,963 WARN [Timer-Driven Process Thread-2]
> o.a.nifi.processors.kafka.KafkaPublisher Timed out while waiting for acks
> from Kafka
> 2016-05-26 16:27:00,963 WARN [Timer-Driven Process Thread-2]
> o.apache.nifi.processors.kafka.PutKafka
> PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] Timed out while waiting
> for acks from Kafka
> 2016-05-26 16:27:00,963 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.processors.kafka.PutKafka
> PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55]
> PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] failed to process due to
> java.lang.NullPointerException; rolling back session:
> java.lang.NullPointerException
> 2016-05-26 16:27:00,965 ERROR [Timer-Driven Process Thread-2]
> o.apache.nifi.processors.kafka.PutKafka
> java.lang.NullPointerException: null
> at java.lang.String.<init>(Unknown Source) ~[na:1.8.0_91]
> at
> org.apache.nifi.processors.kafka.PutKafka.buildFailedFlowFileAttributes(PutKafka.java:396)
> ~[na:na]
> at org.apache.nifi.processors.kafka.PutKafka.onTrigger(PutKafka.java:308)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-0.6.1.jar:0.6.1]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
> [nifi-framework-core-0.6.1.jar:0.6.1]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.6.1.jar:0.6.1]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.6.1.jar:0.6.1]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
> [nifi-framework-core-0.6.1.jar:0.6.1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> [na:1.8.0_91]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source) [na:1.8.0_91]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) [na:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.8.0_91]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.8.0_91]
> at java.lang.Thread.run(Unknown Source) [na:1.8.0_91]
>
> On Thu, May 26, 2016 at 4:28 PM, Oleg Zhurakousky <
> ozhurakousky@hortonworks.com> wrote:
>
>> That is because the processor is already in the process of being stopped
>> and you probably tried to start it again.
>> That said, the fact that you see it is probably a bug since you should
>> not be able to see ‘start’ action until processor is fully stopped.
>>
>> Could you please share more details as to what are you doing (in terms of
>> sequence of events)
>>
>> Thanks
>> Oleg
>>
>> > On May 26, 2016, at 4:12 PM, Igor Kravzov <ig...@gmail.com>
>> wrote:
>> >
>> > Why I am getting warning bellow?
>> > <image.png>
>> >
>> >
>> >
>>
>>
>
>
Re: PutKafka warning - why?
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Igor
Thanks a lot
That’s clearly a bug, simply because NPE originates form our code (must never happen). Thankfully it was fixed already and will be available in 0.7 release. But if you brave enough you can build from master and use it in your current environment in the meantime (let me know if you need help with it).
Otherwise, the 0.7 is not to far away.
Cheers
Oleg
On May 26, 2016, at 4:39 PM, Igor Kravzov <ig...@gmail.com>> wrote:
Actually I wanted to write new email, but will ask here.
I am trying to use PutKafka with Hortonworks HDP 2.4. Latest. The template is simple and shown bellow. The warning above I received when tried to change processor's configuration and start again.
<image.png>
Actually I am unable to send file to Kafka. Getting exception bellow.
2016-05-26 16:25:56,227 INFO [StandardProcessScheduler Thread-4] o.a.k.clients.producer.ProducerConfig ProducerConfig values:
compression.type = none
metric.reporters = []
metadata.max.age.ms<http://metadata.max.age.ms/> = 300000
metadata.fetch.timeout.ms<http://metadata.fetch.timeout.ms/> = 30000
acks = 0
batch.size = 1
reconnect.backoff.ms<http://reconnect.backoff.ms/> = 10
bootstrap.servers = [10.1.132.113:6667<http://10.1.132.113:6667/>]
receive.buffer.bytes = 32768
retry.backoff.ms<http://retry.backoff.ms/> = 100
buffer.memory = 5242880
timeout.ms<http://timeout.ms/> = 30000
key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
retries = 0
max.request.size = 1048576
block.on.buffer.full = true
value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
metrics.sample.window.ms<http://metrics.sample.window.ms/> = 30000
send.buffer.bytes = 131072
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
linger.ms<http://linger.ms/> = 5000
client.id<http://client.id/> = MyTEst
2016-05-26 16:25:56,443 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@6e7a6be5 // Another save pending = false
2016-05-26 16:26:00,771 WARN [StandardProcessScheduler Thread-4] o.a.k.clients.producer.ProducerConfig The configuration partitioner.class = null was supplied but isn't a known config.
2016-05-26 16:26:00,771 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] to run with 1 threads
2016-05-26 16:26:04,979 INFO [pool-16-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2016-05-26 16:26:05,216 INFO [pool-16-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@1227013b checkpointed with 1 Records and 0 Swap Files in 237 milliseconds (Stop-the-world time = 110 milliseconds, Clear Edit Logs time = 124 millis), max Transaction ID 252557730
2016-05-26 16:26:05,216 INFO [pool-16-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 1 records in 237 milliseconds
2016-05-26 16:27:00,963 WARN [Timer-Driven Process Thread-2] o.a.nifi.processors.kafka.KafkaPublisher Timed out while waiting for acks from Kafka
2016-05-26 16:27:00,963 WARN [Timer-Driven Process Thread-2] o.apache.nifi.processors.kafka.PutKafka PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] Timed out while waiting for acks from Kafka
2016-05-26 16:27:00,963 ERROR [Timer-Driven Process Thread-2] o.apache.nifi.processors.kafka.PutKafka PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] failed to process due to java.lang.NullPointerException; rolling back session: java.lang.NullPointerException
2016-05-26 16:27:00,965 ERROR [Timer-Driven Process Thread-2] o.apache.nifi.processors.kafka.PutKafka
java.lang.NullPointerException: null
at java.lang.String.<init>(Unknown Source) ~[na:1.8.0_91]
at org.apache.nifi.processors.kafka.PutKafka.buildFailedFlowFileAttributes(PutKafka.java:396) ~[na:na]
at org.apache.nifi.processors.kafka.PutKafka.onTrigger(PutKafka.java:308) ~[na:na]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.6.1.jar:0.6.1]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059) [nifi-framework-core-0.6.1.jar:0.6.1]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-0.6.1.jar:0.6.1]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-0.6.1.jar:0.6.1]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123) [nifi-framework-core-0.6.1.jar:0.6.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.8.0_91]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_91]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_91]
On Thu, May 26, 2016 at 4:28 PM, Oleg Zhurakousky <oz...@hortonworks.com>> wrote:
That is because the processor is already in the process of being stopped and you probably tried to start it again.
That said, the fact that you see it is probably a bug since you should not be able to see ‘start’ action until processor is fully stopped.
Could you please share more details as to what are you doing (in terms of sequence of events)
Thanks
Oleg
> On May 26, 2016, at 4:12 PM, Igor Kravzov <ig...@gmail.com>> wrote:
>
> Why I am getting warning bellow?
> <image.png>
>
>
>
Re: PutKafka warning - why?
Posted by Igor Kravzov <ig...@gmail.com>.
Actually I wanted to write new email, but will ask here.
I am trying to use PutKafka with Hortonworks HDP 2.4. Latest. The template
is simple and shown bellow. The warning above I received when tried to
change processor's configuration and start again.
[image: Inline image 1]
Actually I am unable to send file to Kafka. Getting exception bellow.
2016-05-26 16:25:56,227 INFO [StandardProcessScheduler Thread-4]
o.a.k.clients.producer.ProducerConfig ProducerConfig values:
compression.type = none
metric.reporters = []
metadata.max.age.ms = 300000
metadata.fetch.timeout.ms = 30000
acks = 0
batch.size = 1
reconnect.backoff.ms = 10
bootstrap.servers = [10.1.132.113:6667]
receive.buffer.bytes = 32768
retry.backoff.ms = 100
buffer.memory = 5242880
timeout.ms = 30000
key.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
retries = 0
max.request.size = 1048576
block.on.buffer.full = true
value.serializer = class
org.apache.kafka.common.serialization.ByteArraySerializer
metrics.sample.window.ms = 30000
send.buffer.bytes = 131072
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
linger.ms = 5000
client.id = MyTEst
2016-05-26 16:25:56,443 INFO [Flow Service Tasks Thread-2]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@6e7a6be5 // Another save pending
= false
2016-05-26 16:26:00,771 WARN [StandardProcessScheduler Thread-4]
o.a.k.clients.producer.ProducerConfig The configuration partitioner.class =
null was supplied but isn't a known config.
2016-05-26 16:26:00,771 INFO [StandardProcessScheduler Thread-8]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] to run with 1 threads
2016-05-26 16:26:04,979 INFO [pool-16-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
2016-05-26 16:26:05,216 INFO [pool-16-thread-1]
org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@1227013b checkpointed with 1 Records
and 0 Swap Files in 237 milliseconds (Stop-the-world time = 110
milliseconds, Clear Edit Logs time = 124 millis), max Transaction ID
252557730
2016-05-26 16:26:05,216 INFO [pool-16-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 1 records in 237 milliseconds
2016-05-26 16:27:00,963 WARN [Timer-Driven Process Thread-2]
o.a.nifi.processors.kafka.KafkaPublisher Timed out while waiting for acks
from Kafka
2016-05-26 16:27:00,963 WARN [Timer-Driven Process Thread-2]
o.apache.nifi.processors.kafka.PutKafka
PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] Timed out while waiting
for acks from Kafka
2016-05-26 16:27:00,963 ERROR [Timer-Driven Process Thread-2]
o.apache.nifi.processors.kafka.PutKafka
PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55]
PutKafka[id=cfe1c4de-ac82-4d7e-86db-44fa0f151c55] failed to process due to
java.lang.NullPointerException; rolling back session:
java.lang.NullPointerException
2016-05-26 16:27:00,965 ERROR [Timer-Driven Process Thread-2]
o.apache.nifi.processors.kafka.PutKafka
java.lang.NullPointerException: null
at java.lang.String.<init>(Unknown Source) ~[na:1.8.0_91]
at
org.apache.nifi.processors.kafka.PutKafka.buildFailedFlowFileAttributes(PutKafka.java:396)
~[na:na]
at org.apache.nifi.processors.kafka.PutKafka.onTrigger(PutKafka.java:308)
~[na:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-0.6.1.jar:0.6.1]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
[nifi-framework-core-0.6.1.jar:0.6.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.6.1.jar:0.6.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.6.1.jar:0.6.1]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
[nifi-framework-core-0.6.1.jar:0.6.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_91]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source) [na:1.8.0_91]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[na:1.8.0_91]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_91]
On Thu, May 26, 2016 at 4:28 PM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:
> That is because the processor is already in the process of being stopped
> and you probably tried to start it again.
> That said, the fact that you see it is probably a bug since you should not
> be able to see ‘start’ action until processor is fully stopped.
>
> Could you please share more details as to what are you doing (in terms of
> sequence of events)
>
> Thanks
> Oleg
>
> > On May 26, 2016, at 4:12 PM, Igor Kravzov <ig...@gmail.com>
> wrote:
> >
> > Why I am getting warning bellow?
> > <image.png>
> >
> >
> >
>
>
Re: PutKafka warning - why?
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
That is because the processor is already in the process of being stopped and you probably tried to start it again.
That said, the fact that you see it is probably a bug since you should not be able to see ‘start’ action until processor is fully stopped.
Could you please share more details as to what are you doing (in terms of sequence of events)
Thanks
Oleg
> On May 26, 2016, at 4:12 PM, Igor Kravzov <ig...@gmail.com> wrote:
>
> Why I am getting warning bellow?
> <image.png>
>
>
>