You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Thiago Rahal Disposti <th...@kryptus.com> on 2019/08/06 16:29:29 UTC
Profiler not writing to hbase at the end of the period
Our metron profiler seems to be writing to the hbase profiler table in
random times and values that doesn't seem to make sense.
Anyone already had a similar issue?
We already tried different time periods(5, 10 , 15) without any luck.
profiler.propertiers:
*##### Profiler #####*
*profiler.input.topic=indexing*
*profiler.output.topic=enrichments*
*profiler.period.duration=5*
*profiler.period.duration.units=MINUTES*
*profiler.window.duration=30*
*profiler.window.duration.units=SECONDS*
*profiler.ttl=30*
*profiler.ttl.units=MINUTES*
*profiler.window.lag=1*
*profiler.window.lag.units=MINUTES*
*profiler.max.routes.per.bolt=100000*
*##### HBase #####*
*profiler.hbase.salt.divisor=1000*
*profiler.hbase.table=profiler*
*profiler.hbase.column.family=P*
*profiler.hbase.batch=10*
*profiler.hbase.flush.interval.seconds=30*
*##### Kafka #####*
*kafka.zk=blackbird2.kryptus:2181,blackbird3.kryptus:2181,blackbird1.kryptus:2181*
*kafka.broker=blackbird2.kryptus:6667,blackbird1.kryptus:6667*
*kafka.start=UNCOMMITTED_EARLIEST*
*kafka.security.protocol=PLAINTEXT*
Config:
[image: image.png]
Parser config:
[image: image.png]
Output on Stellar:
[image: image.png]
Thanks,
Thiago Rahal
Re: Profiler not writing to hbase at the end of the period
Posted by Nick Allen <ni...@nickallen.org>.
> Here i can see that the two values were written at hbase at 4:03:47 and
4:13:34 PM.
> Both values were written before the current profiler period ends,
probably because of my hbase batch size and/or time interval configuration.
I would not expect this to be the case. The value cannot be calculated
prior to the end of the period, no matter what your batch size is.
Could the clock in HBase be skewed or some other cause? I would confirm
the timing of those writes using the Profiler logs. In the Storm UI,
enable debug or trace logging for the package org.apache.metron.profiler.
The Storm worker logs will then contain a lot of detailed information about
each message that is received and when each profile period is flushed. Do
the logs show the flush occurring prior to the end of the period?
> If i guarantee that the batch size it’s not reachable inside one profiler
period and use a time interval bigger than the time period, the values will
be written to hbase only in the end of the profile period?
The value cannot be calculated prior to the end of the period. The
calculated value may write some time after the end of the period, depending
on your batch size and flush frequencies for HBase, but the same calculated
value will eventually be written.
> Other times i found myself in this situation after increasing the batch
size and time interval the profiler stopped writing values at hbase
When just testing at small scale, then set the batch size to 1 until you've
worked through these issues.
On Wed, Aug 7, 2019 at 12:50 PM Thiago Rahal Disposti <
thiago.rahal@kryptus.com> wrote:
> Tests with PROFILE_VERBOSE:
> [image: image.png]
>
> At this moment i have two periods results:
> First: starts at 4:00 PM and ends at 4:05 PM with value equals 22 for
> the entity 169.254.84.237
> Second: starts at 4:10 PM and ends at 4:15 PM with value equals 7 for
> the entity 169.254.84.237
> *Shouldn't be a period from 4:05 to 4:10?
>
> Hbase:
> [image: image.png]
>
> Here i can see that the two values were written at hbase at 4:03:47 and
> 4:13:34 PM.
> Both values were written before the current profiler period ends, probably
> because of my hbase batch size and/or time interval configuration.
> If i guarantee that the batch size it’s not reachable inside one profiler
> period and use a time interval bigger than the time period, the values will
> be written to hbase only in the end of the profile period?
> It's that true? Other times i found my self in this situation after
> increasing the batch size and time interval the profiler stopped writing
> values at hbase
>
> The entity 169.254.84.237 it’s the only one that exists for this profiler.
>
> Thanks,
> Thiago Rahal
>
>
> On Tue, Aug 6, 2019 at 5:40 PM Nick Allen <ni...@nickallen.org> wrote:
>
>> Since it is working in the REPL, but not in Storm, are you sure that the
>> Profiler topology is keeping up with your incoming telemetry? Checkout the
>> Profiler topology in the Storm UI and make sure that it is keeping up.
>>
>> On Tue, Aug 6, 2019 at 3:28 PM Thiago Rahal Disposti <
>> thiago.rahal@kryptus.com> wrote:
>>
>>> Hi Nick,
>>>
>>> (1) That's right, i changed what i was trying to do for something more
>>> simple after things started to go wrong and forget to change the name of
>>> the profiler.
>>> (2) Yes, all tests in the REPL were successful.
>>> (3) Thanks a lot! I can see that the PROFILE_VERBOSE will be extremely
>>> useful.
>>> (4) Exactly
>>>
>>> I will run more tests now and try to figure something out.
>>>
>>> Thanks for the response,
>>> Thiago Rahal
>>>
>>>
>>>
>>>
>>>
>>> <http://www.kryptus.com.br>
>>> *Thiago Rahal *
>>> Cybersecurity
>>>
>>> +55 (19) 3112-5000
>>> thiago.rahal@kryptus.com
>>>
>>> www.kryptus.com <http://www.kryptus.com.br>
>>>
>>> <http://www.kryptus.com.br>
>>>
>>>
>>>
>>> On Tue, Aug 6, 2019 at 4:06 PM Nick Allen <ni...@nickallen.org> wrote:
>>>
>>>> Hi Thiago -
>>>>
>>>> (1) Your profile definition seems to just be counting the number of
>>>> messages that have a certain IP in the ip_src_addr field.
>>>> That logic does not seem to match your profile name which implies you
>>>> are trying to count the number of login failures.
>>>>
>>>> (2) Have you tested your profile in the REPL with PROFILER_INIT and
>>>> PROFILER_APPLY? That should let you validate
>>>> that your profile definition is indeed counting login failures.
>>>>
>>>> (3) You can use `PROFILE_VERBOSE` to retrieve those same profile
>>>> measurements, but with more detailed information
>>>> for each, including the time period of each value. That might help you
>>>> troubleshoot in the REPL a bit more.
>>>>
>>>> (4) I would suggest just setting the value of
>>>> profiler.client.period.duration and profiler.client.period.duration.units
>>>> in your
>>>> global configuration in Zookeeper, rather than passing those values
>>>> into each call that you make to PROFILE_GET. But
>>>> perhaps you already know that and are just troubleshooting your
>>>> current problem.
>>>>
>>>> Outside of those suggestions, I don't quite understand what your
>>>> expectations are and why the values that you are seeing differ.
>>>>
>>>> Best
>>>>
>>>> On Tue, Aug 6, 2019 at 12:30 PM Thiago Rahal Disposti <
>>>> thiago.rahal@kryptus.com> wrote:
>>>>
>>>>> Our metron profiler seems to be writing to the hbase profiler table in
>>>>> random times and values that doesn't seem to make sense.
>>>>>
>>>>> Anyone already had a similar issue?
>>>>>
>>>>> We already tried different time periods(5, 10 , 15) without any luck.
>>>>>
>>>>> profiler.propertiers:
>>>>>
>>>>> *##### Profiler #####*
>>>>>
>>>>>
>>>>> *profiler.input.topic=indexing*
>>>>>
>>>>> *profiler.output.topic=enrichments*
>>>>>
>>>>> *profiler.period.duration=5*
>>>>>
>>>>> *profiler.period.duration.units=MINUTES*
>>>>>
>>>>> *profiler.window.duration=30*
>>>>>
>>>>> *profiler.window.duration.units=SECONDS*
>>>>>
>>>>> *profiler.ttl=30*
>>>>>
>>>>> *profiler.ttl.units=MINUTES*
>>>>>
>>>>> *profiler.window.lag=1*
>>>>>
>>>>> *profiler.window.lag.units=MINUTES*
>>>>>
>>>>> *profiler.max.routes.per.bolt=100000*
>>>>>
>>>>>
>>>>> *##### HBase #####*
>>>>>
>>>>>
>>>>> *profiler.hbase.salt.divisor=1000*
>>>>>
>>>>> *profiler.hbase.table=profiler*
>>>>>
>>>>> *profiler.hbase.column.family=P*
>>>>>
>>>>> *profiler.hbase.batch=10*
>>>>>
>>>>> *profiler.hbase.flush.interval.seconds=30*
>>>>>
>>>>>
>>>>> *##### Kafka #####*
>>>>>
>>>>>
>>>>>
>>>>> *kafka.zk=blackbird2.kryptus:2181,blackbird3.kryptus:2181,blackbird1.kryptus:2181*
>>>>>
>>>>> *kafka.broker=blackbird2.kryptus:6667,blackbird1.kryptus:6667*
>>>>>
>>>>> *kafka.start=UNCOMMITTED_EARLIEST*
>>>>>
>>>>> *kafka.security.protocol=PLAINTEXT*
>>>>>
>>>>>
>>>>> Config:
>>>>> [image: image.png]
>>>>>
>>>>> Parser config:
>>>>> [image: image.png]
>>>>>
>>>>> Output on Stellar:
>>>>> [image: image.png]
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Thiago Rahal
>>>>>
>>>>>
>>>>>
Re: Profiler not writing to hbase at the end of the period
Posted by Thiago Rahal Disposti <th...@kryptus.com>.
Tests with PROFILE_VERBOSE:
[image: image.png]
At this moment i have two periods results:
First: starts at 4:00 PM and ends at 4:05 PM with value equals 22 for the
entity 169.254.84.237
Second: starts at 4:10 PM and ends at 4:15 PM with value equals 7 for the
entity 169.254.84.237
*Shouldn't be a period from 4:05 to 4:10?
Hbase:
[image: image.png]
Here i can see that the two values were written at hbase at 4:03:47 and
4:13:34 PM.
Both values were written before the current profiler period ends, probably
because of my hbase batch size and/or time interval configuration.
If i guarantee that the batch size it’s not reachable inside one profiler
period and use a time interval bigger than the time period, the values will
be written to hbase only in the end of the profile period?
It's that true? Other times i found my self in this situation after
increasing the batch size and time interval the profiler stopped writing
values at hbase
The entity 169.254.84.237 it’s the only one that exists for this profiler.
Thanks,
Thiago Rahal
On Tue, Aug 6, 2019 at 5:40 PM Nick Allen <ni...@nickallen.org> wrote:
> Since it is working in the REPL, but not in Storm, are you sure that the
> Profiler topology is keeping up with your incoming telemetry? Checkout the
> Profiler topology in the Storm UI and make sure that it is keeping up.
>
> On Tue, Aug 6, 2019 at 3:28 PM Thiago Rahal Disposti <
> thiago.rahal@kryptus.com> wrote:
>
>> Hi Nick,
>>
>> (1) That's right, i changed what i was trying to do for something more
>> simple after things started to go wrong and forget to change the name of
>> the profiler.
>> (2) Yes, all tests in the REPL were successful.
>> (3) Thanks a lot! I can see that the PROFILE_VERBOSE will be extremely
>> useful.
>> (4) Exactly
>>
>> I will run more tests now and try to figure something out.
>>
>> Thanks for the response,
>> Thiago Rahal
>>
>>
>>
>>
>>
>> <http://www.kryptus.com.br>
>> *Thiago Rahal *
>> Cybersecurity
>>
>> +55 (19) 3112-5000
>> thiago.rahal@kryptus.com
>>
>> www.kryptus.com <http://www.kryptus.com.br>
>>
>> <http://www.kryptus.com.br>
>>
>>
>>
>> On Tue, Aug 6, 2019 at 4:06 PM Nick Allen <ni...@nickallen.org> wrote:
>>
>>> Hi Thiago -
>>>
>>> (1) Your profile definition seems to just be counting the number of
>>> messages that have a certain IP in the ip_src_addr field.
>>> That logic does not seem to match your profile name which implies you
>>> are trying to count the number of login failures.
>>>
>>> (2) Have you tested your profile in the REPL with PROFILER_INIT and
>>> PROFILER_APPLY? That should let you validate
>>> that your profile definition is indeed counting login failures.
>>>
>>> (3) You can use `PROFILE_VERBOSE` to retrieve those same profile
>>> measurements, but with more detailed information
>>> for each, including the time period of each value. That might help you
>>> troubleshoot in the REPL a bit more.
>>>
>>> (4) I would suggest just setting the value of
>>> profiler.client.period.duration and profiler.client.period.duration.units
>>> in your
>>> global configuration in Zookeeper, rather than passing those values into
>>> each call that you make to PROFILE_GET. But
>>> perhaps you already know that and are just troubleshooting your current
>>> problem.
>>>
>>> Outside of those suggestions, I don't quite understand what your
>>> expectations are and why the values that you are seeing differ.
>>>
>>> Best
>>>
>>> On Tue, Aug 6, 2019 at 12:30 PM Thiago Rahal Disposti <
>>> thiago.rahal@kryptus.com> wrote:
>>>
>>>> Our metron profiler seems to be writing to the hbase profiler table in
>>>> random times and values that doesn't seem to make sense.
>>>>
>>>> Anyone already had a similar issue?
>>>>
>>>> We already tried different time periods(5, 10 , 15) without any luck.
>>>>
>>>> profiler.propertiers:
>>>>
>>>> *##### Profiler #####*
>>>>
>>>>
>>>> *profiler.input.topic=indexing*
>>>>
>>>> *profiler.output.topic=enrichments*
>>>>
>>>> *profiler.period.duration=5*
>>>>
>>>> *profiler.period.duration.units=MINUTES*
>>>>
>>>> *profiler.window.duration=30*
>>>>
>>>> *profiler.window.duration.units=SECONDS*
>>>>
>>>> *profiler.ttl=30*
>>>>
>>>> *profiler.ttl.units=MINUTES*
>>>>
>>>> *profiler.window.lag=1*
>>>>
>>>> *profiler.window.lag.units=MINUTES*
>>>>
>>>> *profiler.max.routes.per.bolt=100000*
>>>>
>>>>
>>>> *##### HBase #####*
>>>>
>>>>
>>>> *profiler.hbase.salt.divisor=1000*
>>>>
>>>> *profiler.hbase.table=profiler*
>>>>
>>>> *profiler.hbase.column.family=P*
>>>>
>>>> *profiler.hbase.batch=10*
>>>>
>>>> *profiler.hbase.flush.interval.seconds=30*
>>>>
>>>>
>>>> *##### Kafka #####*
>>>>
>>>>
>>>>
>>>> *kafka.zk=blackbird2.kryptus:2181,blackbird3.kryptus:2181,blackbird1.kryptus:2181*
>>>>
>>>> *kafka.broker=blackbird2.kryptus:6667,blackbird1.kryptus:6667*
>>>>
>>>> *kafka.start=UNCOMMITTED_EARLIEST*
>>>>
>>>> *kafka.security.protocol=PLAINTEXT*
>>>>
>>>>
>>>> Config:
>>>> [image: image.png]
>>>>
>>>> Parser config:
>>>> [image: image.png]
>>>>
>>>> Output on Stellar:
>>>> [image: image.png]
>>>>
>>>>
>>>> Thanks,
>>>> Thiago Rahal
>>>>
>>>>
>>>>
Re: Profiler not writing to hbase at the end of the period
Posted by Nick Allen <ni...@nickallen.org>.
Since it is working in the REPL, but not in Storm, are you sure that the
Profiler topology is keeping up with your incoming telemetry? Checkout the
Profiler topology in the Storm UI and make sure that it is keeping up.
On Tue, Aug 6, 2019 at 3:28 PM Thiago Rahal Disposti <
thiago.rahal@kryptus.com> wrote:
> Hi Nick,
>
> (1) That's right, i changed what i was trying to do for something more
> simple after things started to go wrong and forget to change the name of
> the profiler.
> (2) Yes, all tests in the REPL were successful.
> (3) Thanks a lot! I can see that the PROFILE_VERBOSE will be extremely
> useful.
> (4) Exactly
>
> I will run more tests now and try to figure something out.
>
> Thanks for the response,
> Thiago Rahal
>
>
>
>
>
> <http://www.kryptus.com.br>
> *Thiago Rahal *
> Cybersecurity
>
> +55 (19) 3112-5000
> thiago.rahal@kryptus.com
>
> www.kryptus.com <http://www.kryptus.com.br>
>
> <http://www.kryptus.com.br>
>
>
>
> On Tue, Aug 6, 2019 at 4:06 PM Nick Allen <ni...@nickallen.org> wrote:
>
>> Hi Thiago -
>>
>> (1) Your profile definition seems to just be counting the number of
>> messages that have a certain IP in the ip_src_addr field.
>> That logic does not seem to match your profile name which implies you are
>> trying to count the number of login failures.
>>
>> (2) Have you tested your profile in the REPL with PROFILER_INIT and
>> PROFILER_APPLY? That should let you validate
>> that your profile definition is indeed counting login failures.
>>
>> (3) You can use `PROFILE_VERBOSE` to retrieve those same profile
>> measurements, but with more detailed information
>> for each, including the time period of each value. That might help you
>> troubleshoot in the REPL a bit more.
>>
>> (4) I would suggest just setting the value of
>> profiler.client.period.duration and profiler.client.period.duration.units
>> in your
>> global configuration in Zookeeper, rather than passing those values into
>> each call that you make to PROFILE_GET. But
>> perhaps you already know that and are just troubleshooting your current
>> problem.
>>
>> Outside of those suggestions, I don't quite understand what your
>> expectations are and why the values that you are seeing differ.
>>
>> Best
>>
>> On Tue, Aug 6, 2019 at 12:30 PM Thiago Rahal Disposti <
>> thiago.rahal@kryptus.com> wrote:
>>
>>> Our metron profiler seems to be writing to the hbase profiler table in
>>> random times and values that doesn't seem to make sense.
>>>
>>> Anyone already had a similar issue?
>>>
>>> We already tried different time periods(5, 10 , 15) without any luck.
>>>
>>> profiler.propertiers:
>>>
>>> *##### Profiler #####*
>>>
>>>
>>> *profiler.input.topic=indexing*
>>>
>>> *profiler.output.topic=enrichments*
>>>
>>> *profiler.period.duration=5*
>>>
>>> *profiler.period.duration.units=MINUTES*
>>>
>>> *profiler.window.duration=30*
>>>
>>> *profiler.window.duration.units=SECONDS*
>>>
>>> *profiler.ttl=30*
>>>
>>> *profiler.ttl.units=MINUTES*
>>>
>>> *profiler.window.lag=1*
>>>
>>> *profiler.window.lag.units=MINUTES*
>>>
>>> *profiler.max.routes.per.bolt=100000*
>>>
>>>
>>> *##### HBase #####*
>>>
>>>
>>> *profiler.hbase.salt.divisor=1000*
>>>
>>> *profiler.hbase.table=profiler*
>>>
>>> *profiler.hbase.column.family=P*
>>>
>>> *profiler.hbase.batch=10*
>>>
>>> *profiler.hbase.flush.interval.seconds=30*
>>>
>>>
>>> *##### Kafka #####*
>>>
>>>
>>>
>>> *kafka.zk=blackbird2.kryptus:2181,blackbird3.kryptus:2181,blackbird1.kryptus:2181*
>>>
>>> *kafka.broker=blackbird2.kryptus:6667,blackbird1.kryptus:6667*
>>>
>>> *kafka.start=UNCOMMITTED_EARLIEST*
>>>
>>> *kafka.security.protocol=PLAINTEXT*
>>>
>>>
>>> Config:
>>> [image: image.png]
>>>
>>> Parser config:
>>> [image: image.png]
>>>
>>> Output on Stellar:
>>> [image: image.png]
>>>
>>>
>>> Thanks,
>>> Thiago Rahal
>>>
>>>
>>>
Re: Profiler not writing to hbase at the end of the period
Posted by Thiago Rahal Disposti <th...@kryptus.com>.
Hi Nick,
(1) That's right, i changed what i was trying to do for something more
simple after things started to go wrong and forget to change the name of
the profiler.
(2) Yes, all tests in the REPL were successful.
(3) Thanks a lot! I can see that the PROFILE_VERBOSE will be extremely
useful.
(4) Exactly
I will run more tests now and try to figure something out.
Thanks for the response,
Thiago Rahal
<http://www.kryptus.com.br>
*Thiago Rahal *
Cybersecurity
+55 (19) 3112-5000
thiago.rahal@kryptus.com
www.kryptus.com <http://www.kryptus.com.br>
<http://www.kryptus.com.br>
On Tue, Aug 6, 2019 at 4:06 PM Nick Allen <ni...@nickallen.org> wrote:
> Hi Thiago -
>
> (1) Your profile definition seems to just be counting the number of
> messages that have a certain IP in the ip_src_addr field.
> That logic does not seem to match your profile name which implies you are
> trying to count the number of login failures.
>
> (2) Have you tested your profile in the REPL with PROFILER_INIT and
> PROFILER_APPLY? That should let you validate
> that your profile definition is indeed counting login failures.
>
> (3) You can use `PROFILE_VERBOSE` to retrieve those same profile
> measurements, but with more detailed information
> for each, including the time period of each value. That might help you
> troubleshoot in the REPL a bit more.
>
> (4) I would suggest just setting the value of
> profiler.client.period.duration and profiler.client.period.duration.units
> in your
> global configuration in Zookeeper, rather than passing those values into
> each call that you make to PROFILE_GET. But
> perhaps you already know that and are just troubleshooting your current
> problem.
>
> Outside of those suggestions, I don't quite understand what your
> expectations are and why the values that you are seeing differ.
>
> Best
>
> On Tue, Aug 6, 2019 at 12:30 PM Thiago Rahal Disposti <
> thiago.rahal@kryptus.com> wrote:
>
>> Our metron profiler seems to be writing to the hbase profiler table in
>> random times and values that doesn't seem to make sense.
>>
>> Anyone already had a similar issue?
>>
>> We already tried different time periods(5, 10 , 15) without any luck.
>>
>> profiler.propertiers:
>>
>> *##### Profiler #####*
>>
>>
>> *profiler.input.topic=indexing*
>>
>> *profiler.output.topic=enrichments*
>>
>> *profiler.period.duration=5*
>>
>> *profiler.period.duration.units=MINUTES*
>>
>> *profiler.window.duration=30*
>>
>> *profiler.window.duration.units=SECONDS*
>>
>> *profiler.ttl=30*
>>
>> *profiler.ttl.units=MINUTES*
>>
>> *profiler.window.lag=1*
>>
>> *profiler.window.lag.units=MINUTES*
>>
>> *profiler.max.routes.per.bolt=100000*
>>
>>
>> *##### HBase #####*
>>
>>
>> *profiler.hbase.salt.divisor=1000*
>>
>> *profiler.hbase.table=profiler*
>>
>> *profiler.hbase.column.family=P*
>>
>> *profiler.hbase.batch=10*
>>
>> *profiler.hbase.flush.interval.seconds=30*
>>
>>
>> *##### Kafka #####*
>>
>>
>>
>> *kafka.zk=blackbird2.kryptus:2181,blackbird3.kryptus:2181,blackbird1.kryptus:2181*
>>
>> *kafka.broker=blackbird2.kryptus:6667,blackbird1.kryptus:6667*
>>
>> *kafka.start=UNCOMMITTED_EARLIEST*
>>
>> *kafka.security.protocol=PLAINTEXT*
>>
>>
>> Config:
>> [image: image.png]
>>
>> Parser config:
>> [image: image.png]
>>
>> Output on Stellar:
>> [image: image.png]
>>
>>
>> Thanks,
>> Thiago Rahal
>>
>>
>>
Re: Profiler not writing to hbase at the end of the period
Posted by Nick Allen <ni...@nickallen.org>.
Hi Thiago -
(1) Your profile definition seems to just be counting the number of
messages that have a certain IP in the ip_src_addr field.
That logic does not seem to match your profile name which implies you are
trying to count the number of login failures.
(2) Have you tested your profile in the REPL with PROFILER_INIT and
PROFILER_APPLY? That should let you validate
that your profile definition is indeed counting login failures.
(3) You can use `PROFILE_VERBOSE` to retrieve those same profile
measurements, but with more detailed information
for each, including the time period of each value. That might help you
troubleshoot in the REPL a bit more.
(4) I would suggest just setting the value of
profiler.client.period.duration and profiler.client.period.duration.units
in your
global configuration in Zookeeper, rather than passing those values into
each call that you make to PROFILE_GET. But
perhaps you already know that and are just troubleshooting your current
problem.
Outside of those suggestions, I don't quite understand what your
expectations are and why the values that you are seeing differ.
Best
On Tue, Aug 6, 2019 at 12:30 PM Thiago Rahal Disposti <
thiago.rahal@kryptus.com> wrote:
> Our metron profiler seems to be writing to the hbase profiler table in
> random times and values that doesn't seem to make sense.
>
> Anyone already had a similar issue?
>
> We already tried different time periods(5, 10 , 15) without any luck.
>
> profiler.propertiers:
>
> *##### Profiler #####*
>
>
> *profiler.input.topic=indexing*
>
> *profiler.output.topic=enrichments*
>
> *profiler.period.duration=5*
>
> *profiler.period.duration.units=MINUTES*
>
> *profiler.window.duration=30*
>
> *profiler.window.duration.units=SECONDS*
>
> *profiler.ttl=30*
>
> *profiler.ttl.units=MINUTES*
>
> *profiler.window.lag=1*
>
> *profiler.window.lag.units=MINUTES*
>
> *profiler.max.routes.per.bolt=100000*
>
>
> *##### HBase #####*
>
>
> *profiler.hbase.salt.divisor=1000*
>
> *profiler.hbase.table=profiler*
>
> *profiler.hbase.column.family=P*
>
> *profiler.hbase.batch=10*
>
> *profiler.hbase.flush.interval.seconds=30*
>
>
> *##### Kafka #####*
>
>
>
> *kafka.zk=blackbird2.kryptus:2181,blackbird3.kryptus:2181,blackbird1.kryptus:2181*
>
> *kafka.broker=blackbird2.kryptus:6667,blackbird1.kryptus:6667*
>
> *kafka.start=UNCOMMITTED_EARLIEST*
>
> *kafka.security.protocol=PLAINTEXT*
>
>
> Config:
> [image: image.png]
>
> Parser config:
> [image: image.png]
>
> Output on Stellar:
> [image: image.png]
>
>
> Thanks,
> Thiago Rahal
>
>
>