Profiler debug

[Gonçalo Pedras <go...@ctd.pt>]

Hi,
I'm experiencing some GC overhead limits in my profiler. The profiler works fine while testing in Stellar enviorment, although it fails most of the time as a running topology. My profiler has 1 profile only.
My profile has the following configuration:

·         5 integer counters, when It finds a specific string it adds to the respective counter.

·         Unique list of strings, when It finds a new string it adds to the list.

·         A fixed string field (stays the same throughout the profiler duration).
My profile flushes:

·         Profiler: a hardcoded Json map of the results of the counters. Ex: {\"counter1\": variable1, \"counter2\": variable2 ..}

·         Triage: the integer counters, the fixed string value and the size of the unique string list.
The only reason I have a fixed value on the profile is that I don't want to enrich the same data twice, so I just might pass it through the profiler.
Is there any other way to debug the profiler while running as a topology? Because I can't seem to understand why it ends in error and the storm logs don't help much. I've tweaked some options like the profile period, time to live and worker childopts.

Thanks

Re: Profiler debug

[Nick Allen <ni...@nickallen.org>]

https://storm.apache.org/releases/1.2.3/dynamic-log-level-settings.html

On Fri, Mar 6, 2020 at 9:37 AM Gonçalo Pedras <go...@ctd.pt> wrote:

> Thanks for replying. One more question: how do I set the debug for
> “org.apache.metron.profiler” ?
>
> *Thanks*
>
>
>
>

RE: Profiler debug

[Gonçalo Pedras <go...@ctd.pt>]

Thanks for replying. One more question: how do I set the debug for “org.apache.metron.profiler” ?
Thanks

Re: Profiler debug

[Nick Allen <ni...@nickallen.org>]

Setting DEBUG logging for 'org.apache.metron.profiler' should show you very
detailed information about what is happening with the Profiler in your
topology.

On Fri, Mar 6, 2020 at 7:18 AM Gonçalo Pedras <go...@ctd.pt> wrote:

> Hi,
>
> I’m experiencing some GC overhead limits in my profiler. The profiler
> works fine while testing in Stellar enviorment, although it fails most of
> the time as a running topology. My profiler has 1 profile only.
>
> My profile has the following configuration:
>
> ·         5 integer counters, when It finds a specific string it adds to
> the respective counter.
>
> ·         Unique list of strings, when It finds a new string it adds to
> the list.
>
> ·         A fixed string field (stays the same throughout the profiler
> duration).
>
> My profile flushes:
>
> ·         Profiler: a hardcoded Json map of the results of the counters.
> Ex: {\”counter1\”: variable1, \”counter2\”: variable2 ..}
>
> ·         Triage: the integer counters, the fixed string value and the
> size of the unique string list.
>
> The only reason I have a fixed value on the profile is that I don’t want
> to enrich the same data twice, so I just might pass it through the
> profiler.
>
> Is there any other way to debug the profiler while running as a topology?
> Because I can’t seem to understand why it ends in error and the storm logs
> don’t help much. I’ve tweaked some options like the profile period, time to
> live and worker childopts.
>
>
>
> Thanks
>