question about ConsumeKafka metrics for incoming data

[Dominique De Vito <dd...@gmail.com>]

Hi,

While running a ConsumeKafka processor, I noticed there are metrics for
incoming data, but there was no value (like "bytes in") provided in the UI.

And as far as I have digged into the ConsumeKafka processor, I didn't found
any code related to feeding the metrics related to the incoming data.

Is it true that ConsumeKafka processor provides no value for the metrics
related to the incoming data ?

Is it the same for all processors dealing with data coming from outside
Nifi ?

Thanks.

Regards,
Dominique

Re: question about ConsumeKafka metrics for incoming data

[Bryan Bende <bb...@gmail.com>]

Hello,

Bytes in is the # of bytes read from incoming flow files. Since source
processors don't have incoming flow files they will never have bytes
in or flow files in.

-Bryan
On Thu, Oct 18, 2018 at 11:25 AM Dominique De Vito <dd...@gmail.com> wrote:
>
> Hi,
>
> While running a ConsumeKafka processor, I noticed there are metrics for incoming data, but there was no value (like "bytes in") provided in the UI.
>
> And as far as I have digged into the ConsumeKafka processor, I didn't found any code related to feeding the metrics related to the incoming data.
>
> Is it true that ConsumeKafka processor provides no value for the metrics related to the incoming data ?
>
> Is it the same for all processors dealing with data coming from outside Nifi ?
>
> Thanks.
>
> Regards,
> Dominique
>

Re: question about ConsumeKafka metrics for incoming data

[Dominique De Vito <dd...@gmail.com>]

> The value should be close to how many bytes have been consumed from
Kafka.

Very good hint. Thanks Jeff.

And thanks Bryan too !

Dominique


Le jeu. 18 oct. 2018 à 17:52, Jeff <jt...@gmail.com> a écrit :

> Dominique,
>
> The "bytes in" value represents how many bytes have been received from an
> incoming connection.  Since ConsumeKafka is directly consuming data from
> Kafka (a "source" processor), and not from a connection from an upstream
> processor, it won't show a value for "bytes in".  If you want to see how
> much data ConsumeKafka has sent downstream, you can look at "bytes out".
> The value should be close to how many bytes have been consumed from Kafka.
> I'm not familiar with the Kafka processors, but I assume the messages are
> encrypted/compressed.
>
> On Thu, Oct 18, 2018 at 11:25 AM Dominique De Vito <dd...@gmail.com>
> wrote:
>
>> Hi,
>>
>> While running a ConsumeKafka processor, I noticed there are metrics for
>> incoming data, but there was no value (like "bytes in") provided in the UI.
>>
>> And as far as I have digged into the ConsumeKafka processor, I didn't
>> found any code related to feeding the metrics related to the incoming data.
>>
>> Is it true that ConsumeKafka processor provides no value for the metrics
>> related to the incoming data ?
>>
>> Is it the same for all processors dealing with data coming from outside
>> Nifi ?
>>
>> Thanks.
>>
>> Regards,
>> Dominique
>>
>>

Re: question about ConsumeKafka metrics for incoming data

[Jeff <jt...@gmail.com>]

Dominique,

The "bytes in" value represents how many bytes have been received from an
incoming connection.  Since ConsumeKafka is directly consuming data from
Kafka (a "source" processor), and not from a connection from an upstream
processor, it won't show a value for "bytes in".  If you want to see how
much data ConsumeKafka has sent downstream, you can look at "bytes out".
The value should be close to how many bytes have been consumed from Kafka.
I'm not familiar with the Kafka processors, but I assume the messages are
encrypted/compressed.

On Thu, Oct 18, 2018 at 11:25 AM Dominique De Vito <dd...@gmail.com>
wrote:

> Hi,
>
> While running a ConsumeKafka processor, I noticed there are metrics for
> incoming data, but there was no value (like "bytes in") provided in the UI.
>
> And as far as I have digged into the ConsumeKafka processor, I didn't
> found any code related to feeding the metrics related to the incoming data.
>
> Is it true that ConsumeKafka processor provides no value for the metrics
> related to the incoming data ?
>
> Is it the same for all processors dealing with data coming from outside
> Nifi ?
>
> Thanks.
>
> Regards,
> Dominique
>
>