You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by AJAY GUPTA <aj...@gmail.com> on 2017/01/19 17:25:36 UTC
[DISCUSS] Stram events log levels
Hi Apex community,
Currently, events logged by stram such as StartContainer, StartOperator,
StopContainer, StopOperator, etc dont have any associated priority level.
We can provide log levels for such Stram events. Log levels can be INFO,
WARN, ERROR
Eg:
1. Start Container, Start Operator are INFO level events
2. OperatorError is ERROR level event
3. Stop Container, Stop Operator are WARN level events
Log level for events can help in user experience when showing the event
list in a GUI. eg: In datatorrent gateway UI, we can provide color-coded
log levels so that user can focus more on ERROR and WARN events.
Re: [DISCUSS] Stram events log levels
Posted by AJAY GUPTA <aj...@gmail.com>.
Hi
For the initial release, I am planning to have static event log levels,
i.e., log levels without considering context in which event was generated.
I will add support for updating this log level so that in future, we can
update log levels considering the context of the event.
Ajay
On Fri, Jan 20, 2017 at 2:04 PM, AJAY GUPTA <aj...@gmail.com> wrote:
> Nice point Tushar. I will look into how we can pass on different event log
> levels for same event type.
>
>
> Ajay
>
> On Fri, Jan 20, 2017 at 10:18 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
>> +1 for this feature. The event priority is not statically defained by
>> its type, it also depends on on context in which event is generated.
>> for example STOP_CONTAINER event could be because of
>>
>> - jvm issue/out of memory - This could be at much higher level
>> - All operators in containers have shutdown voluntary/or extra
>> container released - this could be at lower level.
>>
>> - Tushar.
>>
>>
>>
>> On Fri, Jan 20, 2017 at 12:32 AM, Sanjay Pujare <sa...@datatorrent.com>
>> wrote:
>> > Strong +1 for the reasons cited by Ajay and Hitesh.
>> >
>> > Do we really need to modify the EventInfo class? Wouldn’t the
>> BeanUtils.describe(event) automatically populate the EventLogLevel into
>> “data” (a Map<String, String>) as a property? All the downstream code
>> including the REST API will then just return the EventLogLevel as part of
>> JSONObject without making any changes, right?
>> >
>> > Another thing: these levels should ideally match the corresponding
>> entries in the dt.log file as some users might expect.
>> >
>> > On 1/19/17, 10:10 AM, "Hitesh Kapoor" <hi...@datatorrent.com> wrote:
>> >
>> > +1 it will enhance the user experience and the approach suggested
>> by Ajay
>> > doesn't seems to add significant overhead. Also I feel that in
>> future it
>> > can be enhanced further and UI can also filter the event logs based
>> on
>> > these log levels.
>> >
>> > On Jan 19, 2017 10:57 PM, "AJAY GUPTA" <aj...@gmail.com>
>> wrote:
>> >
>> > > Current event log mechanism : The events recorded are currently
>> recorded in
>> > > datatorrent/apps/{appid}/events/ folder in files named
>> part{id}.txt .
>> > > eg
>> > > : datatorrent/apps/application_1483801541352_0003/events/part0
>> .txt
>> > >
>> > > We can access these events via REST Api which returns the list of
>> events as
>> > > List<EventInfo> objects.
>> > >
>> > >
>> > > The below approach can be used for implementing this. Kindly let
>> me know
>> > > your viewpoints/suggestions on the approach.
>> > >
>> > > *Required Changes:*
>> > >
>> > > a) Add a enum field like EventLogLevel in StramEvent, assign the
>> log level
>> > > in constructors of different Stram events like StartOperator,
>> > > StartContainer, StopOperator, etc.
>> > >
>> > > b) Add eventLogLevel in EventInfo class. The events returned via
>> REST api
>> > > from StramClient is a list of EventInfo objects.
>> > >
>> > > c) Changes in EventsAgent - processPartFile() method to parse the
>> > > eventLogLevel logged into the part{id}.txt file
>> > >
>> > >
>> > > On Thu, Jan 19, 2017 at 10:55 PM, AJAY GUPTA <
>> ajaygit158@gmail.com> wrote:
>> > >
>> > > > Hi Apex community,
>> > > >
>> > > >
>> > > > Currently, events logged by stram such as StartContainer,
>> StartOperator,
>> > > > StopContainer, StopOperator, etc dont have any associated
>> priority level.
>> > > >
>> > > > We can provide log levels for such Stram events. Log levels can
>> be INFO,
>> > > > WARN, ERROR
>> > > > Eg:
>> > > > 1. Start Container, Start Operator are INFO level events
>> > > > 2. OperatorError is ERROR level event
>> > > > 3. Stop Container, Stop Operator are WARN level events
>> > > >
>> > > > Log level for events can help in user experience when showing
>> the event
>> > > > list in a GUI. eg: In datatorrent gateway UI, we can provide
>> color-coded
>> > > > log levels so that user can focus more on ERROR and WARN events.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> >
>> >
>>
>
>
Re: [DISCUSS] Stram events log levels
Posted by AJAY GUPTA <aj...@gmail.com>.
Nice point Tushar. I will look into how we can pass on different event log
levels for same event type.
Ajay
On Fri, Jan 20, 2017 at 10:18 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> +1 for this feature. The event priority is not statically defained by
> its type, it also depends on on context in which event is generated.
> for example STOP_CONTAINER event could be because of
>
> - jvm issue/out of memory - This could be at much higher level
> - All operators in containers have shutdown voluntary/or extra
> container released - this could be at lower level.
>
> - Tushar.
>
>
>
> On Fri, Jan 20, 2017 at 12:32 AM, Sanjay Pujare <sa...@datatorrent.com>
> wrote:
> > Strong +1 for the reasons cited by Ajay and Hitesh.
> >
> > Do we really need to modify the EventInfo class? Wouldn’t the
> BeanUtils.describe(event) automatically populate the EventLogLevel into
> “data” (a Map<String, String>) as a property? All the downstream code
> including the REST API will then just return the EventLogLevel as part of
> JSONObject without making any changes, right?
> >
> > Another thing: these levels should ideally match the corresponding
> entries in the dt.log file as some users might expect.
> >
> > On 1/19/17, 10:10 AM, "Hitesh Kapoor" <hi...@datatorrent.com> wrote:
> >
> > +1 it will enhance the user experience and the approach suggested by
> Ajay
> > doesn't seems to add significant overhead. Also I feel that in
> future it
> > can be enhanced further and UI can also filter the event logs based
> on
> > these log levels.
> >
> > On Jan 19, 2017 10:57 PM, "AJAY GUPTA" <aj...@gmail.com> wrote:
> >
> > > Current event log mechanism : The events recorded are currently
> recorded in
> > > datatorrent/apps/{appid}/events/ folder in files named
> part{id}.txt .
> > > eg
> > > : datatorrent/apps/application_1483801541352_0003/events/part0.txt
> > >
> > > We can access these events via REST Api which returns the list of
> events as
> > > List<EventInfo> objects.
> > >
> > >
> > > The below approach can be used for implementing this. Kindly let
> me know
> > > your viewpoints/suggestions on the approach.
> > >
> > > *Required Changes:*
> > >
> > > a) Add a enum field like EventLogLevel in StramEvent, assign the
> log level
> > > in constructors of different Stram events like StartOperator,
> > > StartContainer, StopOperator, etc.
> > >
> > > b) Add eventLogLevel in EventInfo class. The events returned via
> REST api
> > > from StramClient is a list of EventInfo objects.
> > >
> > > c) Changes in EventsAgent - processPartFile() method to parse the
> > > eventLogLevel logged into the part{id}.txt file
> > >
> > >
> > > On Thu, Jan 19, 2017 at 10:55 PM, AJAY GUPTA <aj...@gmail.com>
> wrote:
> > >
> > > > Hi Apex community,
> > > >
> > > >
> > > > Currently, events logged by stram such as StartContainer,
> StartOperator,
> > > > StopContainer, StopOperator, etc dont have any associated
> priority level.
> > > >
> > > > We can provide log levels for such Stram events. Log levels can
> be INFO,
> > > > WARN, ERROR
> > > > Eg:
> > > > 1. Start Container, Start Operator are INFO level events
> > > > 2. OperatorError is ERROR level event
> > > > 3. Stop Container, Stop Operator are WARN level events
> > > >
> > > > Log level for events can help in user experience when showing
> the event
> > > > list in a GUI. eg: In datatorrent gateway UI, we can provide
> color-coded
> > > > log levels so that user can focus more on ERROR and WARN events.
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>
Re: [DISCUSS] Stram events log levels
Posted by Tushar Gosavi <tu...@datatorrent.com>.
+1 for this feature. The event priority is not statically defained by
its type, it also depends on on context in which event is generated.
for example STOP_CONTAINER event could be because of
- jvm issue/out of memory - This could be at much higher level
- All operators in containers have shutdown voluntary/or extra
container released - this could be at lower level.
- Tushar.
On Fri, Jan 20, 2017 at 12:32 AM, Sanjay Pujare <sa...@datatorrent.com> wrote:
> Strong +1 for the reasons cited by Ajay and Hitesh.
>
> Do we really need to modify the EventInfo class? Wouldn’t the BeanUtils.describe(event) automatically populate the EventLogLevel into “data” (a Map<String, String>) as a property? All the downstream code including the REST API will then just return the EventLogLevel as part of JSONObject without making any changes, right?
>
> Another thing: these levels should ideally match the corresponding entries in the dt.log file as some users might expect.
>
> On 1/19/17, 10:10 AM, "Hitesh Kapoor" <hi...@datatorrent.com> wrote:
>
> +1 it will enhance the user experience and the approach suggested by Ajay
> doesn't seems to add significant overhead. Also I feel that in future it
> can be enhanced further and UI can also filter the event logs based on
> these log levels.
>
> On Jan 19, 2017 10:57 PM, "AJAY GUPTA" <aj...@gmail.com> wrote:
>
> > Current event log mechanism : The events recorded are currently recorded in
> > datatorrent/apps/{appid}/events/ folder in files named part{id}.txt .
> > eg
> > : datatorrent/apps/application_1483801541352_0003/events/part0.txt
> >
> > We can access these events via REST Api which returns the list of events as
> > List<EventInfo> objects.
> >
> >
> > The below approach can be used for implementing this. Kindly let me know
> > your viewpoints/suggestions on the approach.
> >
> > *Required Changes:*
> >
> > a) Add a enum field like EventLogLevel in StramEvent, assign the log level
> > in constructors of different Stram events like StartOperator,
> > StartContainer, StopOperator, etc.
> >
> > b) Add eventLogLevel in EventInfo class. The events returned via REST api
> > from StramClient is a list of EventInfo objects.
> >
> > c) Changes in EventsAgent - processPartFile() method to parse the
> > eventLogLevel logged into the part{id}.txt file
> >
> >
> > On Thu, Jan 19, 2017 at 10:55 PM, AJAY GUPTA <aj...@gmail.com> wrote:
> >
> > > Hi Apex community,
> > >
> > >
> > > Currently, events logged by stram such as StartContainer, StartOperator,
> > > StopContainer, StopOperator, etc dont have any associated priority level.
> > >
> > > We can provide log levels for such Stram events. Log levels can be INFO,
> > > WARN, ERROR
> > > Eg:
> > > 1. Start Container, Start Operator are INFO level events
> > > 2. OperatorError is ERROR level event
> > > 3. Stop Container, Stop Operator are WARN level events
> > >
> > > Log level for events can help in user experience when showing the event
> > > list in a GUI. eg: In datatorrent gateway UI, we can provide color-coded
> > > log levels so that user can focus more on ERROR and WARN events.
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
Re: [DISCUSS] Stram events log levels
Posted by Sanjay Pujare <sa...@datatorrent.com>.
Strong +1 for the reasons cited by Ajay and Hitesh.
Do we really need to modify the EventInfo class? Wouldn’t the BeanUtils.describe(event) automatically populate the EventLogLevel into “data” (a Map<String, String>) as a property? All the downstream code including the REST API will then just return the EventLogLevel as part of JSONObject without making any changes, right?
Another thing: these levels should ideally match the corresponding entries in the dt.log file as some users might expect.
On 1/19/17, 10:10 AM, "Hitesh Kapoor" <hi...@datatorrent.com> wrote:
+1 it will enhance the user experience and the approach suggested by Ajay
doesn't seems to add significant overhead. Also I feel that in future it
can be enhanced further and UI can also filter the event logs based on
these log levels.
On Jan 19, 2017 10:57 PM, "AJAY GUPTA" <aj...@gmail.com> wrote:
> Current event log mechanism : The events recorded are currently recorded in
> datatorrent/apps/{appid}/events/ folder in files named part{id}.txt .
> eg
> : datatorrent/apps/application_1483801541352_0003/events/part0.txt
>
> We can access these events via REST Api which returns the list of events as
> List<EventInfo> objects.
>
>
> The below approach can be used for implementing this. Kindly let me know
> your viewpoints/suggestions on the approach.
>
> *Required Changes:*
>
> a) Add a enum field like EventLogLevel in StramEvent, assign the log level
> in constructors of different Stram events like StartOperator,
> StartContainer, StopOperator, etc.
>
> b) Add eventLogLevel in EventInfo class. The events returned via REST api
> from StramClient is a list of EventInfo objects.
>
> c) Changes in EventsAgent - processPartFile() method to parse the
> eventLogLevel logged into the part{id}.txt file
>
>
> On Thu, Jan 19, 2017 at 10:55 PM, AJAY GUPTA <aj...@gmail.com> wrote:
>
> > Hi Apex community,
> >
> >
> > Currently, events logged by stram such as StartContainer, StartOperator,
> > StopContainer, StopOperator, etc dont have any associated priority level.
> >
> > We can provide log levels for such Stram events. Log levels can be INFO,
> > WARN, ERROR
> > Eg:
> > 1. Start Container, Start Operator are INFO level events
> > 2. OperatorError is ERROR level event
> > 3. Stop Container, Stop Operator are WARN level events
> >
> > Log level for events can help in user experience when showing the event
> > list in a GUI. eg: In datatorrent gateway UI, we can provide color-coded
> > log levels so that user can focus more on ERROR and WARN events.
> >
> >
> >
> >
> >
>
Re: [DISCUSS] Stram events log levels
Posted by Hitesh Kapoor <hi...@datatorrent.com>.
+1 it will enhance the user experience and the approach suggested by Ajay
doesn't seems to add significant overhead. Also I feel that in future it
can be enhanced further and UI can also filter the event logs based on
these log levels.
On Jan 19, 2017 10:57 PM, "AJAY GUPTA" <aj...@gmail.com> wrote:
> Current event log mechanism : The events recorded are currently recorded in
> datatorrent/apps/{appid}/events/ folder in files named part{id}.txt .
> eg
> : datatorrent/apps/application_1483801541352_0003/events/part0.txt
>
> We can access these events via REST Api which returns the list of events as
> List<EventInfo> objects.
>
>
> The below approach can be used for implementing this. Kindly let me know
> your viewpoints/suggestions on the approach.
>
> *Required Changes:*
>
> a) Add a enum field like EventLogLevel in StramEvent, assign the log level
> in constructors of different Stram events like StartOperator,
> StartContainer, StopOperator, etc.
>
> b) Add eventLogLevel in EventInfo class. The events returned via REST api
> from StramClient is a list of EventInfo objects.
>
> c) Changes in EventsAgent - processPartFile() method to parse the
> eventLogLevel logged into the part{id}.txt file
>
>
> On Thu, Jan 19, 2017 at 10:55 PM, AJAY GUPTA <aj...@gmail.com> wrote:
>
> > Hi Apex community,
> >
> >
> > Currently, events logged by stram such as StartContainer, StartOperator,
> > StopContainer, StopOperator, etc dont have any associated priority level.
> >
> > We can provide log levels for such Stram events. Log levels can be INFO,
> > WARN, ERROR
> > Eg:
> > 1. Start Container, Start Operator are INFO level events
> > 2. OperatorError is ERROR level event
> > 3. Stop Container, Stop Operator are WARN level events
> >
> > Log level for events can help in user experience when showing the event
> > list in a GUI. eg: In datatorrent gateway UI, we can provide color-coded
> > log levels so that user can focus more on ERROR and WARN events.
> >
> >
> >
> >
> >
>
Re: [DISCUSS] Stram events log levels
Posted by AJAY GUPTA <aj...@gmail.com>.
Current event log mechanism : The events recorded are currently recorded in
datatorrent/apps/{appid}/events/ folder in files named part{id}.txt . eg
: datatorrent/apps/application_1483801541352_0003/events/part0.txt
We can access these events via REST Api which returns the list of events as
List<EventInfo> objects.
The below approach can be used for implementing this. Kindly let me know
your viewpoints/suggestions on the approach.
*Required Changes:*
a) Add a enum field like EventLogLevel in StramEvent, assign the log level
in constructors of different Stram events like StartOperator,
StartContainer, StopOperator, etc.
b) Add eventLogLevel in EventInfo class. The events returned via REST api
from StramClient is a list of EventInfo objects.
c) Changes in EventsAgent - processPartFile() method to parse the
eventLogLevel logged into the part{id}.txt file
On Thu, Jan 19, 2017 at 10:55 PM, AJAY GUPTA <aj...@gmail.com> wrote:
> Hi Apex community,
>
>
> Currently, events logged by stram such as StartContainer, StartOperator,
> StopContainer, StopOperator, etc dont have any associated priority level.
>
> We can provide log levels for such Stram events. Log levels can be INFO,
> WARN, ERROR
> Eg:
> 1. Start Container, Start Operator are INFO level events
> 2. OperatorError is ERROR level event
> 3. Stop Container, Stop Operator are WARN level events
>
> Log level for events can help in user experience when showing the event
> list in a GUI. eg: In datatorrent gateway UI, we can provide color-coded
> log levels so that user can focus more on ERROR and WARN events.
>
>
>
>
>