You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Priyanka Gugale <pr...@datatorrent.com> on 2016/11/22 18:00:13 UTC
Adding new log4j appender to Apex core
Hi,
I am working on APEXCORE-563
<https://issues.apache.org/jira/browse/APEXCORE-563>
As per this Jira we should put log file name in container/operator events.
The problem is current RollingFileAppender keeps renaming files from 1 to 2
to ... n as files reach maximum allowed file size. Because of constant
renaming of files we can't put a fixed file name in stram event.
To overcome this I would like to add a new log4j appender to ApexCore.
There are two ways I can implement this:
1. Have Daily rolling file appender. The current file will be recognized
based on timestamp in file name. Also to control max file size, we need to
keep rolling files based on size as well.
2. Have Rolling File Appender but do not rename files. When max file size
is reached create new file with next number e.g. crate log file dt.log.2
after dt.log.1 is full. Also to recognize the latest file keep the softlink
named dt.log pointing to current log file.
I would prefer to implement approach 2. Please provide your
comments/feedback if you feel otherwise.
Also to use this new appender we need to use our custom log4j.properties
file instead of one present in hadoop conf. For that we need to set jvm
option -Dlog4j.configuration. I am planning to update file dt-site.xml in
folder ~/.dt and default properties file available in apex archetype to
set jvm options as follows:
<property>
<name>dt.attr.CONTAINER_JVM_OPTIONS</name>
<value>-Dlog4j.configuration=log4j.props</value>
</property>
And I will copy log4j.props file in ~/.dt folder as well as apex-archetypes.
Lastly if someone still miss this new log4j properties file or jvm option
to set -Dlog4j.configuration we will not put log file name in event raised
by container or operator.
Please provide your feedback on this approach.
-Priyanka
Re: Adding new log4j appender to Apex core
Posted by Vlad Rozov <v....@datatorrent.com>.
David,
Yes, I understand that the functionality will only be available in case
of the predefined log file appender. IMO, such assumption is too
restrictive and it does not look that the default configuration will
cover majority of use cases. Also, JIRA talks about providing offset in
the log file. Does this mean that events can only be consumed by a tool?
For human line number will be more useful.
Thank you,
Vlad
On 11/23/16 22:27, David Yan wrote:
> Vlad,
>
> The feature only works *if the log file name at error does not change later*
> .
>
> In this case, Priyanka proposes a default appender that the user can use
> that has this behavior while log rotation is still supported.
>
> If the user has a custom log appender, the feature can still work if it
> satisfies the above requirement. Otherwise, there is no way for this
> feature to work.
>
> We can support a configuration option to force the inclusion of log
> location and offset in the error STRAM event even if the Apex appender is
> not used, if the user knows that they are using a custom appender that
> satisfies the requirement.
>
> David
>
> On Wed, Nov 23, 2016 at 8:47 PM, Vlad Rozov <v....@datatorrent.com> wrote:
>
>> Additionally, I think that it is necessary to re-evaluate the
>> requirements. Custom logging is quite common and many enterprises/Devops
>> have own preferences/policy for log rotation and logging format. I saw
>> instances when logging was redirected to stdout. By enforcing specific
>> rotation policy or log format, the feature is more likely not to be used.
>>
>> Thank you,
>>
>> Vlad
>>
>>
>> On 11/23/16 14:21, Vlad Rozov wrote:
>>
>>> Both approaches look quite "hacky" to me.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 11/23/16 00:01, Mohit Jotwani wrote:
>>>
>>>> +1 - Approach 2
>>>>
>>>> Regards,
>>>> Mohit
>>>>
>>>> On Wed, Nov 23, 2016 at 12:35 PM, AJAY GUPTA <aj...@gmail.com>
>>>> wrote:
>>>>
>>>> +1 for approach 2.
>>>>>
>>>>> Regards,
>>>>> Ajay
>>>>>
>>>>> On Wed, Nov 23, 2016 at 12:16 PM, David Yan <da...@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> The goal of this log4j appender is to provide a log offset and the fixed
>>>>>> name of the container log file (instead of apex.log becoming apex.log.1
>>>>>>
>>>>> and
>>>>>
>>>>>> then apex.log.2, etc due to rotation) as part of an error STRAM event
>>>>>> so
>>>>>> users can easily locate the log entries around the error.
>>>>>>
>>>>>> The user can override the appender, but in that case, the engine
>>>>>> detects
>>>>>> that and will not include the log location as part of the STRAM event.
>>>>>>
>>>>>> David
>>>>>>
>>>>>> On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <
>>>>>>
>>>>> priyanka@datatorrent.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>> Thomas,
>>>>>>> Yes log4j is ultimately owned by user, and they should be able to
>>>>>>>
>>>>>> override
>>>>>>
>>>>>>> it. What I am trying to do is provide a default behavior for Apex. In
>>>>>>>
>>>>>> case
>>>>>>
>>>>>>> user isn't using any logger of their own we should use this new
>>>>>>>
>>>>>> appender
>>>>>> of
>>>>>>
>>>>>>> Apex rather than using standard log4j appender as per hadoop config.
>>>>>>>
>>>>>>> Sanjay,
>>>>>>> Archetype is the good place to put this and I will add it there, but
>>>>>>>
>>>>>> many
>>>>>> time people won't use it. So I wanted to keep it at ~/.dt as well. Is
>>>>>> there
>>>>>>
>>>>>>> any other default config folder for Apex?
>>>>>>>
>>>>>>> Also I am not relying on anything. If we fail to find config in app
>>>>>>> jar
>>>>>>>
>>>>>> or
>>>>>>
>>>>>>> ~/.dt we are going to skip usage of this new appender.
>>>>>>>
>>>>>>> -Priyanka
>>>>>>>
>>>>>>> On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <
>>>>>>> sanjay@datatorrent.com
>>>>>>> wrote:
>>>>>>>
>>>>>>> The only way to \u201cenforce\u201d this new appender is to update the
>>>>>>> archetypes
>>>>>> (apex-app-archetype and apex-conf-archetype under apex-core/ ) to
>>>>>>> use
>>>>>> the
>>>>>>>> new ones as default. But there does not seem to be a way to enforce
>>>>>>>>
>>>>>>> this
>>>>>>> for anyone not using the archetypes.
>>>>>>>> I agree with not relying on ~/.dt in apex-core.
>>>>>>>>
>>>>>>>> On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
>>>>>>>>
>>>>>>>> The log4j configuration is ultimately owned by the user, so how
>>>>>>>>
>>>>>>> do
>>>>>> you
>>>>>>>> want
>>>>>>>> to enforce a custom appender?
>>>>>>>>
>>>>>>>> I don't think that this should rely on anything in ~/.dt either
>>>>>>>>
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>> On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
>>>>>>>> priyanka@datatorrent.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > I am working on APEXCORE-563
>>>>>>>> > <https://issues.apache.org/jira/browse/APEXCORE-563>
>>>>>>>> > As per this Jira we should put log file name in
>>>>>>>>
>>>>>>> container/operator
>>>>>>> events.
>>>>>>>> > The problem is current RollingFileAppender keeps renaming
>>>>>>>> files
>>>>>>>>
>>>>>>> from
>>>>>>>
>>>>>>>> 1 to 2
>>>>>>>> > to ... n as files reach maximum allowed file size. Because of
>>>>>>>> constant
>>>>>>>> > renaming of files we can't put a fixed file name in stram
>>>>>>>>
>>>>>>> event.
>>>>>> >
>>>>>>>> > To overcome this I would like to add a new log4j appender to
>>>>>>>> ApexCore.
>>>>>>>> > There are two ways I can implement this:
>>>>>>>> > 1. Have Daily rolling file appender. The current file will be
>>>>>>>> recognized
>>>>>>>> > based on timestamp in file name. Also to control max file
>>>>>>>> size,
>>>>>>>>
>>>>>>> we
>>>>>>> need to
>>>>>>>> > keep rolling files based on size as well.
>>>>>>>> > 2. Have Rolling File Appender but do not rename files. When
>>>>>>>> max
>>>>>>>>
>>>>>>> file
>>>>>>>
>>>>>>>> size
>>>>>>>> > is reached create new file with next number e.g. crate log
>>>>>>>> file
>>>>>>>> dt.log.2
>>>>>>>> > after dt.log.1 is full. Also to recognize the latest file keep
>>>>>>>>
>>>>>>> the
>>>>>>> softlink
>>>>>>>> > named dt.log pointing to current log file.
>>>>>>>> >
>>>>>>>> > I would prefer to implement approach 2. Please provide your
>>>>>>>> > comments/feedback if you feel otherwise.
>>>>>>>> >
>>>>>>>> > Also to use this new appender we need to use our custom
>>>>>>>> log4j.properties
>>>>>>>> > file instead of one present in hadoop conf. For that we need
>>>>>>>> to
>>>>>>>>
>>>>>>> set
>>>>>>> jvm
>>>>>>>> > option -Dlog4j.configuration. I am planning to update file
>>>>>>>> dt-site.xml in
>>>>>>>> > folder ~/.dt and default properties file available in apex
>>>>>>>> archetype to
>>>>>>>> > set jvm options as follows:
>>>>>>>> > <property>
>>>>>>>> > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
>>>>>>>> > <value>-Dlog4j.configuration=log4j.props</value>
>>>>>>>> > </property>
>>>>>>>> >
>>>>>>>> > And I will copy log4j.props file in ~/.dt folder as well as
>>>>>>>> > apex-archetypes.
>>>>>>>> >
>>>>>>>> > Lastly if someone still miss this new log4j properties file or
>>>>>>>>
>>>>>>> jvm
>>>>>>> option
>>>>>>>> > to set -Dlog4j.configuration we will not put log file name in
>>>>>>>>
>>>>>>> event
>>>>>>> raised
>>>>>>>> > by container or operator.
>>>>>>>> >
>>>>>>>> > Please provide your feedback on this approach.
>>>>>>>> >
>>>>>>>> > -Priyanka
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: Adding new log4j appender to Apex core
Posted by David Yan <da...@datatorrent.com>.
Vlad,
The feature only works *if the log file name at error does not change later*
.
In this case, Priyanka proposes a default appender that the user can use
that has this behavior while log rotation is still supported.
If the user has a custom log appender, the feature can still work if it
satisfies the above requirement. Otherwise, there is no way for this
feature to work.
We can support a configuration option to force the inclusion of log
location and offset in the error STRAM event even if the Apex appender is
not used, if the user knows that they are using a custom appender that
satisfies the requirement.
David
On Wed, Nov 23, 2016 at 8:47 PM, Vlad Rozov <v....@datatorrent.com> wrote:
> Additionally, I think that it is necessary to re-evaluate the
> requirements. Custom logging is quite common and many enterprises/Devops
> have own preferences/policy for log rotation and logging format. I saw
> instances when logging was redirected to stdout. By enforcing specific
> rotation policy or log format, the feature is more likely not to be used.
>
> Thank you,
>
> Vlad
>
>
> On 11/23/16 14:21, Vlad Rozov wrote:
>
>> Both approaches look quite "hacky" to me.
>>
>> Thank you,
>>
>> Vlad
>>
>> On 11/23/16 00:01, Mohit Jotwani wrote:
>>
>>> +1 - Approach 2
>>>
>>> Regards,
>>> Mohit
>>>
>>> On Wed, Nov 23, 2016 at 12:35 PM, AJAY GUPTA <aj...@gmail.com>
>>> wrote:
>>>
>>> +1 for approach 2.
>>>>
>>>>
>>>> Regards,
>>>> Ajay
>>>>
>>>> On Wed, Nov 23, 2016 at 12:16 PM, David Yan <da...@datatorrent.com>
>>>> wrote:
>>>>
>>>> The goal of this log4j appender is to provide a log offset and the fixed
>>>>> name of the container log file (instead of apex.log becoming apex.log.1
>>>>>
>>>> and
>>>>
>>>>> then apex.log.2, etc due to rotation) as part of an error STRAM event
>>>>> so
>>>>> users can easily locate the log entries around the error.
>>>>>
>>>>> The user can override the appender, but in that case, the engine
>>>>> detects
>>>>> that and will not include the log location as part of the STRAM event.
>>>>>
>>>>> David
>>>>>
>>>>> On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <
>>>>>
>>>> priyanka@datatorrent.com
>>>>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> Thomas,
>>>>>> Yes log4j is ultimately owned by user, and they should be able to
>>>>>>
>>>>> override
>>>>>
>>>>>> it. What I am trying to do is provide a default behavior for Apex. In
>>>>>>
>>>>> case
>>>>>
>>>>>> user isn't using any logger of their own we should use this new
>>>>>>
>>>>> appender
>>>>
>>>>> of
>>>>>
>>>>>> Apex rather than using standard log4j appender as per hadoop config.
>>>>>>
>>>>>> Sanjay,
>>>>>> Archetype is the good place to put this and I will add it there, but
>>>>>>
>>>>> many
>>>>
>>>>> time people won't use it. So I wanted to keep it at ~/.dt as well. Is
>>>>>>
>>>>> there
>>>>>
>>>>>> any other default config folder for Apex?
>>>>>>
>>>>>> Also I am not relying on anything. If we fail to find config in app
>>>>>> jar
>>>>>>
>>>>> or
>>>>>
>>>>>> ~/.dt we are going to skip usage of this new appender.
>>>>>>
>>>>>> -Priyanka
>>>>>>
>>>>>> On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <
>>>>>> sanjay@datatorrent.com
>>>>>> wrote:
>>>>>>
>>>>>> The only way to “enforce” this new appender is to update the
>>>>>>>
>>>>>> archetypes
>>>>
>>>>> (apex-app-archetype and apex-conf-archetype under apex-core/ ) to
>>>>>>>
>>>>>> use
>>>>
>>>>> the
>>>>>>
>>>>>>> new ones as default. But there does not seem to be a way to enforce
>>>>>>>
>>>>>> this
>>>>>
>>>>>> for anyone not using the archetypes.
>>>>>>>
>>>>>>> I agree with not relying on ~/.dt in apex-core.
>>>>>>>
>>>>>>> On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
>>>>>>>
>>>>>>> The log4j configuration is ultimately owned by the user, so how
>>>>>>>
>>>>>> do
>>>>
>>>>> you
>>>>>>
>>>>>>> want
>>>>>>> to enforce a custom appender?
>>>>>>>
>>>>>>> I don't think that this should rely on anything in ~/.dt either
>>>>>>>
>>>>>>> Thomas
>>>>>>>
>>>>>>> On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
>>>>>>> priyanka@datatorrent.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi,
>>>>>>> >
>>>>>>> > I am working on APEXCORE-563
>>>>>>> > <https://issues.apache.org/jira/browse/APEXCORE-563>
>>>>>>> > As per this Jira we should put log file name in
>>>>>>>
>>>>>> container/operator
>>>>>
>>>>>> events.
>>>>>>> > The problem is current RollingFileAppender keeps renaming
>>>>>>> files
>>>>>>>
>>>>>> from
>>>>>>
>>>>>>> 1 to 2
>>>>>>> > to ... n as files reach maximum allowed file size. Because of
>>>>>>> constant
>>>>>>> > renaming of files we can't put a fixed file name in stram
>>>>>>>
>>>>>> event.
>>>>
>>>>> >
>>>>>>> > To overcome this I would like to add a new log4j appender to
>>>>>>> ApexCore.
>>>>>>> > There are two ways I can implement this:
>>>>>>> > 1. Have Daily rolling file appender. The current file will be
>>>>>>> recognized
>>>>>>> > based on timestamp in file name. Also to control max file
>>>>>>> size,
>>>>>>>
>>>>>> we
>>>>>
>>>>>> need to
>>>>>>> > keep rolling files based on size as well.
>>>>>>> > 2. Have Rolling File Appender but do not rename files. When
>>>>>>> max
>>>>>>>
>>>>>> file
>>>>>>
>>>>>>> size
>>>>>>> > is reached create new file with next number e.g. crate log
>>>>>>> file
>>>>>>> dt.log.2
>>>>>>> > after dt.log.1 is full. Also to recognize the latest file keep
>>>>>>>
>>>>>> the
>>>>>
>>>>>> softlink
>>>>>>> > named dt.log pointing to current log file.
>>>>>>> >
>>>>>>> > I would prefer to implement approach 2. Please provide your
>>>>>>> > comments/feedback if you feel otherwise.
>>>>>>> >
>>>>>>> > Also to use this new appender we need to use our custom
>>>>>>> log4j.properties
>>>>>>> > file instead of one present in hadoop conf. For that we need
>>>>>>> to
>>>>>>>
>>>>>> set
>>>>>
>>>>>> jvm
>>>>>>> > option -Dlog4j.configuration. I am planning to update file
>>>>>>> dt-site.xml in
>>>>>>> > folder ~/.dt and default properties file available in apex
>>>>>>> archetype to
>>>>>>> > set jvm options as follows:
>>>>>>> > <property>
>>>>>>> > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
>>>>>>> > <value>-Dlog4j.configuration=log4j.props</value>
>>>>>>> > </property>
>>>>>>> >
>>>>>>> > And I will copy log4j.props file in ~/.dt folder as well as
>>>>>>> > apex-archetypes.
>>>>>>> >
>>>>>>> > Lastly if someone still miss this new log4j properties file or
>>>>>>>
>>>>>> jvm
>>>>>
>>>>>> option
>>>>>>> > to set -Dlog4j.configuration we will not put log file name in
>>>>>>>
>>>>>> event
>>>>>
>>>>>> raised
>>>>>>> > by container or operator.
>>>>>>> >
>>>>>>> > Please provide your feedback on this approach.
>>>>>>> >
>>>>>>> > -Priyanka
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>
>
Re: Adding new log4j appender to Apex core
Posted by Vlad Rozov <v....@datatorrent.com>.
Additionally, I think that it is necessary to re-evaluate the
requirements. Custom logging is quite common and many enterprises/Devops
have own preferences/policy for log rotation and logging format. I saw
instances when logging was redirected to stdout. By enforcing specific
rotation policy or log format, the feature is more likely not to be used.
Thank you,
Vlad
On 11/23/16 14:21, Vlad Rozov wrote:
> Both approaches look quite "hacky" to me.
>
> Thank you,
>
> Vlad
>
> On 11/23/16 00:01, Mohit Jotwani wrote:
>> +1 - Approach 2
>>
>> Regards,
>> Mohit
>>
>> On Wed, Nov 23, 2016 at 12:35 PM, AJAY GUPTA <aj...@gmail.com>
>> wrote:
>>
>>> +1 for approach 2.
>>>
>>>
>>> Regards,
>>> Ajay
>>>
>>> On Wed, Nov 23, 2016 at 12:16 PM, David Yan <da...@datatorrent.com>
>>> wrote:
>>>
>>>> The goal of this log4j appender is to provide a log offset and the
>>>> fixed
>>>> name of the container log file (instead of apex.log becoming
>>>> apex.log.1
>>> and
>>>> then apex.log.2, etc due to rotation) as part of an error STRAM
>>>> event so
>>>> users can easily locate the log entries around the error.
>>>>
>>>> The user can override the appender, but in that case, the engine
>>>> detects
>>>> that and will not include the log location as part of the STRAM event.
>>>>
>>>> David
>>>>
>>>> On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <
>>> priyanka@datatorrent.com
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thomas,
>>>>> Yes log4j is ultimately owned by user, and they should be able to
>>>> override
>>>>> it. What I am trying to do is provide a default behavior for Apex. In
>>>> case
>>>>> user isn't using any logger of their own we should use this new
>>> appender
>>>> of
>>>>> Apex rather than using standard log4j appender as per hadoop config.
>>>>>
>>>>> Sanjay,
>>>>> Archetype is the good place to put this and I will add it there, but
>>> many
>>>>> time people won't use it. So I wanted to keep it at ~/.dt as well. Is
>>>> there
>>>>> any other default config folder for Apex?
>>>>>
>>>>> Also I am not relying on anything. If we fail to find config in
>>>>> app jar
>>>> or
>>>>> ~/.dt we are going to skip usage of this new appender.
>>>>>
>>>>> -Priyanka
>>>>>
>>>>> On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare
>>>>> <sanjay@datatorrent.com
>>>>> wrote:
>>>>>
>>>>>> The only way to \u201cenforce\u201d this new appender is to update the
>>> archetypes
>>>>>> (apex-app-archetype and apex-conf-archetype under apex-core/ ) to
>>> use
>>>>> the
>>>>>> new ones as default. But there does not seem to be a way to enforce
>>>> this
>>>>>> for anyone not using the archetypes.
>>>>>>
>>>>>> I agree with not relying on ~/.dt in apex-core.
>>>>>>
>>>>>> On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
>>>>>>
>>>>>> The log4j configuration is ultimately owned by the user, so how
>>> do
>>>>> you
>>>>>> want
>>>>>> to enforce a custom appender?
>>>>>>
>>>>>> I don't think that this should rely on anything in ~/.dt either
>>>>>>
>>>>>> Thomas
>>>>>>
>>>>>> On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
>>>>>> priyanka@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > I am working on APEXCORE-563
>>>>>> > <https://issues.apache.org/jira/browse/APEXCORE-563>
>>>>>> > As per this Jira we should put log file name in
>>>> container/operator
>>>>>> events.
>>>>>> > The problem is current RollingFileAppender keeps renaming
>>>>>> files
>>>>> from
>>>>>> 1 to 2
>>>>>> > to ... n as files reach maximum allowed file size. Because of
>>>>>> constant
>>>>>> > renaming of files we can't put a fixed file name in stram
>>> event.
>>>>>> >
>>>>>> > To overcome this I would like to add a new log4j appender to
>>>>>> ApexCore.
>>>>>> > There are two ways I can implement this:
>>>>>> > 1. Have Daily rolling file appender. The current file will be
>>>>>> recognized
>>>>>> > based on timestamp in file name. Also to control max file
>>>>>> size,
>>>> we
>>>>>> need to
>>>>>> > keep rolling files based on size as well.
>>>>>> > 2. Have Rolling File Appender but do not rename files.
>>>>>> When max
>>>>> file
>>>>>> size
>>>>>> > is reached create new file with next number e.g. crate log
>>>>>> file
>>>>>> dt.log.2
>>>>>> > after dt.log.1 is full. Also to recognize the latest file
>>>>>> keep
>>>> the
>>>>>> softlink
>>>>>> > named dt.log pointing to current log file.
>>>>>> >
>>>>>> > I would prefer to implement approach 2. Please provide your
>>>>>> > comments/feedback if you feel otherwise.
>>>>>> >
>>>>>> > Also to use this new appender we need to use our custom
>>>>>> log4j.properties
>>>>>> > file instead of one present in hadoop conf. For that we
>>>>>> need to
>>>> set
>>>>>> jvm
>>>>>> > option -Dlog4j.configuration. I am planning to update file
>>>>>> dt-site.xml in
>>>>>> > folder ~/.dt and default properties file available in apex
>>>>>> archetype to
>>>>>> > set jvm options as follows:
>>>>>> > <property>
>>>>>> > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
>>>>>> > <value>-Dlog4j.configuration=log4j.props</value>
>>>>>> > </property>
>>>>>> >
>>>>>> > And I will copy log4j.props file in ~/.dt folder as well as
>>>>>> > apex-archetypes.
>>>>>> >
>>>>>> > Lastly if someone still miss this new log4j properties
>>>>>> file or
>>>> jvm
>>>>>> option
>>>>>> > to set -Dlog4j.configuration we will not put log file name in
>>>> event
>>>>>> raised
>>>>>> > by container or operator.
>>>>>> >
>>>>>> > Please provide your feedback on this approach.
>>>>>> >
>>>>>> > -Priyanka
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>
Re: Adding new log4j appender to Apex core
Posted by Vlad Rozov <v....@datatorrent.com>.
Both approaches look quite "hacky" to me.
Thank you,
Vlad
On 11/23/16 00:01, Mohit Jotwani wrote:
> +1 - Approach 2
>
> Regards,
> Mohit
>
> On Wed, Nov 23, 2016 at 12:35 PM, AJAY GUPTA <aj...@gmail.com> wrote:
>
>> +1 for approach 2.
>>
>>
>> Regards,
>> Ajay
>>
>> On Wed, Nov 23, 2016 at 12:16 PM, David Yan <da...@datatorrent.com> wrote:
>>
>>> The goal of this log4j appender is to provide a log offset and the fixed
>>> name of the container log file (instead of apex.log becoming apex.log.1
>> and
>>> then apex.log.2, etc due to rotation) as part of an error STRAM event so
>>> users can easily locate the log entries around the error.
>>>
>>> The user can override the appender, but in that case, the engine detects
>>> that and will not include the log location as part of the STRAM event.
>>>
>>> David
>>>
>>> On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <
>> priyanka@datatorrent.com
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thomas,
>>>> Yes log4j is ultimately owned by user, and they should be able to
>>> override
>>>> it. What I am trying to do is provide a default behavior for Apex. In
>>> case
>>>> user isn't using any logger of their own we should use this new
>> appender
>>> of
>>>> Apex rather than using standard log4j appender as per hadoop config.
>>>>
>>>> Sanjay,
>>>> Archetype is the good place to put this and I will add it there, but
>> many
>>>> time people won't use it. So I wanted to keep it at ~/.dt as well. Is
>>> there
>>>> any other default config folder for Apex?
>>>>
>>>> Also I am not relying on anything. If we fail to find config in app jar
>>> or
>>>> ~/.dt we are going to skip usage of this new appender.
>>>>
>>>> -Priyanka
>>>>
>>>> On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <sanjay@datatorrent.com
>>>> wrote:
>>>>
>>>>> The only way to \u201cenforce\u201d this new appender is to update the
>> archetypes
>>>>> (apex-app-archetype and apex-conf-archetype under apex-core/ ) to
>> use
>>>> the
>>>>> new ones as default. But there does not seem to be a way to enforce
>>> this
>>>>> for anyone not using the archetypes.
>>>>>
>>>>> I agree with not relying on ~/.dt in apex-core.
>>>>>
>>>>> On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
>>>>>
>>>>> The log4j configuration is ultimately owned by the user, so how
>> do
>>>> you
>>>>> want
>>>>> to enforce a custom appender?
>>>>>
>>>>> I don't think that this should rely on anything in ~/.dt either
>>>>>
>>>>> Thomas
>>>>>
>>>>> On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
>>>>> priyanka@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > I am working on APEXCORE-563
>>>>> > <https://issues.apache.org/jira/browse/APEXCORE-563>
>>>>> > As per this Jira we should put log file name in
>>> container/operator
>>>>> events.
>>>>> > The problem is current RollingFileAppender keeps renaming files
>>>> from
>>>>> 1 to 2
>>>>> > to ... n as files reach maximum allowed file size. Because of
>>>>> constant
>>>>> > renaming of files we can't put a fixed file name in stram
>> event.
>>>>> >
>>>>> > To overcome this I would like to add a new log4j appender to
>>>>> ApexCore.
>>>>> > There are two ways I can implement this:
>>>>> > 1. Have Daily rolling file appender. The current file will be
>>>>> recognized
>>>>> > based on timestamp in file name. Also to control max file size,
>>> we
>>>>> need to
>>>>> > keep rolling files based on size as well.
>>>>> > 2. Have Rolling File Appender but do not rename files. When max
>>>> file
>>>>> size
>>>>> > is reached create new file with next number e.g. crate log file
>>>>> dt.log.2
>>>>> > after dt.log.1 is full. Also to recognize the latest file keep
>>> the
>>>>> softlink
>>>>> > named dt.log pointing to current log file.
>>>>> >
>>>>> > I would prefer to implement approach 2. Please provide your
>>>>> > comments/feedback if you feel otherwise.
>>>>> >
>>>>> > Also to use this new appender we need to use our custom
>>>>> log4j.properties
>>>>> > file instead of one present in hadoop conf. For that we need to
>>> set
>>>>> jvm
>>>>> > option -Dlog4j.configuration. I am planning to update file
>>>>> dt-site.xml in
>>>>> > folder ~/.dt and default properties file available in apex
>>>>> archetype to
>>>>> > set jvm options as follows:
>>>>> > <property>
>>>>> > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
>>>>> > <value>-Dlog4j.configuration=log4j.props</value>
>>>>> > </property>
>>>>> >
>>>>> > And I will copy log4j.props file in ~/.dt folder as well as
>>>>> > apex-archetypes.
>>>>> >
>>>>> > Lastly if someone still miss this new log4j properties file or
>>> jvm
>>>>> option
>>>>> > to set -Dlog4j.configuration we will not put log file name in
>>> event
>>>>> raised
>>>>> > by container or operator.
>>>>> >
>>>>> > Please provide your feedback on this approach.
>>>>> >
>>>>> > -Priyanka
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>>
Re: Adding new log4j appender to Apex core
Posted by Mohit Jotwani <mo...@datatorrent.com>.
+1 - Approach 2
Regards,
Mohit
On Wed, Nov 23, 2016 at 12:35 PM, AJAY GUPTA <aj...@gmail.com> wrote:
> +1 for approach 2.
>
>
> Regards,
> Ajay
>
> On Wed, Nov 23, 2016 at 12:16 PM, David Yan <da...@datatorrent.com> wrote:
>
> > The goal of this log4j appender is to provide a log offset and the fixed
> > name of the container log file (instead of apex.log becoming apex.log.1
> and
> > then apex.log.2, etc due to rotation) as part of an error STRAM event so
> > users can easily locate the log entries around the error.
> >
> > The user can override the appender, but in that case, the engine detects
> > that and will not include the log location as part of the STRAM event.
> >
> > David
> >
> > On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <
> priyanka@datatorrent.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > Thomas,
> > > Yes log4j is ultimately owned by user, and they should be able to
> > override
> > > it. What I am trying to do is provide a default behavior for Apex. In
> > case
> > > user isn't using any logger of their own we should use this new
> appender
> > of
> > > Apex rather than using standard log4j appender as per hadoop config.
> > >
> > > Sanjay,
> > > Archetype is the good place to put this and I will add it there, but
> many
> > > time people won't use it. So I wanted to keep it at ~/.dt as well. Is
> > there
> > > any other default config folder for Apex?
> > >
> > > Also I am not relying on anything. If we fail to find config in app jar
> > or
> > > ~/.dt we are going to skip usage of this new appender.
> > >
> > > -Priyanka
> > >
> > > On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <sanjay@datatorrent.com
> >
> > > wrote:
> > >
> > > > The only way to “enforce” this new appender is to update the
> archetypes
> > > > (apex-app-archetype and apex-conf-archetype under apex-core/ ) to
> use
> > > the
> > > > new ones as default. But there does not seem to be a way to enforce
> > this
> > > > for anyone not using the archetypes.
> > > >
> > > > I agree with not relying on ~/.dt in apex-core.
> > > >
> > > > On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
> > > >
> > > > The log4j configuration is ultimately owned by the user, so how
> do
> > > you
> > > > want
> > > > to enforce a custom appender?
> > > >
> > > > I don't think that this should rely on anything in ~/.dt either
> > > >
> > > > Thomas
> > > >
> > > > On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> > > > priyanka@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am working on APEXCORE-563
> > > > > <https://issues.apache.org/jira/browse/APEXCORE-563>
> > > > > As per this Jira we should put log file name in
> > container/operator
> > > > events.
> > > > > The problem is current RollingFileAppender keeps renaming files
> > > from
> > > > 1 to 2
> > > > > to ... n as files reach maximum allowed file size. Because of
> > > > constant
> > > > > renaming of files we can't put a fixed file name in stram
> event.
> > > > >
> > > > > To overcome this I would like to add a new log4j appender to
> > > > ApexCore.
> > > > > There are two ways I can implement this:
> > > > > 1. Have Daily rolling file appender. The current file will be
> > > > recognized
> > > > > based on timestamp in file name. Also to control max file size,
> > we
> > > > need to
> > > > > keep rolling files based on size as well.
> > > > > 2. Have Rolling File Appender but do not rename files. When max
> > > file
> > > > size
> > > > > is reached create new file with next number e.g. crate log file
> > > > dt.log.2
> > > > > after dt.log.1 is full. Also to recognize the latest file keep
> > the
> > > > softlink
> > > > > named dt.log pointing to current log file.
> > > > >
> > > > > I would prefer to implement approach 2. Please provide your
> > > > > comments/feedback if you feel otherwise.
> > > > >
> > > > > Also to use this new appender we need to use our custom
> > > > log4j.properties
> > > > > file instead of one present in hadoop conf. For that we need to
> > set
> > > > jvm
> > > > > option -Dlog4j.configuration. I am planning to update file
> > > > dt-site.xml in
> > > > > folder ~/.dt and default properties file available in apex
> > > > archetype to
> > > > > set jvm options as follows:
> > > > > <property>
> > > > > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
> > > > > <value>-Dlog4j.configuration=log4j.props</value>
> > > > > </property>
> > > > >
> > > > > And I will copy log4j.props file in ~/.dt folder as well as
> > > > > apex-archetypes.
> > > > >
> > > > > Lastly if someone still miss this new log4j properties file or
> > jvm
> > > > option
> > > > > to set -Dlog4j.configuration we will not put log file name in
> > event
> > > > raised
> > > > > by container or operator.
> > > > >
> > > > > Please provide your feedback on this approach.
> > > > >
> > > > > -Priyanka
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: Adding new log4j appender to Apex core
Posted by AJAY GUPTA <aj...@gmail.com>.
+1 for approach 2.
Regards,
Ajay
On Wed, Nov 23, 2016 at 12:16 PM, David Yan <da...@datatorrent.com> wrote:
> The goal of this log4j appender is to provide a log offset and the fixed
> name of the container log file (instead of apex.log becoming apex.log.1 and
> then apex.log.2, etc due to rotation) as part of an error STRAM event so
> users can easily locate the log entries around the error.
>
> The user can override the appender, but in that case, the engine detects
> that and will not include the log location as part of the STRAM event.
>
> David
>
> On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <priyanka@datatorrent.com
> >
> wrote:
>
> > Hi,
> >
> > Thomas,
> > Yes log4j is ultimately owned by user, and they should be able to
> override
> > it. What I am trying to do is provide a default behavior for Apex. In
> case
> > user isn't using any logger of their own we should use this new appender
> of
> > Apex rather than using standard log4j appender as per hadoop config.
> >
> > Sanjay,
> > Archetype is the good place to put this and I will add it there, but many
> > time people won't use it. So I wanted to keep it at ~/.dt as well. Is
> there
> > any other default config folder for Apex?
> >
> > Also I am not relying on anything. If we fail to find config in app jar
> or
> > ~/.dt we are going to skip usage of this new appender.
> >
> > -Priyanka
> >
> > On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <sa...@datatorrent.com>
> > wrote:
> >
> > > The only way to “enforce” this new appender is to update the archetypes
> > > (apex-app-archetype and apex-conf-archetype under apex-core/ ) to use
> > the
> > > new ones as default. But there does not seem to be a way to enforce
> this
> > > for anyone not using the archetypes.
> > >
> > > I agree with not relying on ~/.dt in apex-core.
> > >
> > > On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
> > >
> > > The log4j configuration is ultimately owned by the user, so how do
> > you
> > > want
> > > to enforce a custom appender?
> > >
> > > I don't think that this should rely on anything in ~/.dt either
> > >
> > > Thomas
> > >
> > > On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> > > priyanka@datatorrent.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am working on APEXCORE-563
> > > > <https://issues.apache.org/jira/browse/APEXCORE-563>
> > > > As per this Jira we should put log file name in
> container/operator
> > > events.
> > > > The problem is current RollingFileAppender keeps renaming files
> > from
> > > 1 to 2
> > > > to ... n as files reach maximum allowed file size. Because of
> > > constant
> > > > renaming of files we can't put a fixed file name in stram event.
> > > >
> > > > To overcome this I would like to add a new log4j appender to
> > > ApexCore.
> > > > There are two ways I can implement this:
> > > > 1. Have Daily rolling file appender. The current file will be
> > > recognized
> > > > based on timestamp in file name. Also to control max file size,
> we
> > > need to
> > > > keep rolling files based on size as well.
> > > > 2. Have Rolling File Appender but do not rename files. When max
> > file
> > > size
> > > > is reached create new file with next number e.g. crate log file
> > > dt.log.2
> > > > after dt.log.1 is full. Also to recognize the latest file keep
> the
> > > softlink
> > > > named dt.log pointing to current log file.
> > > >
> > > > I would prefer to implement approach 2. Please provide your
> > > > comments/feedback if you feel otherwise.
> > > >
> > > > Also to use this new appender we need to use our custom
> > > log4j.properties
> > > > file instead of one present in hadoop conf. For that we need to
> set
> > > jvm
> > > > option -Dlog4j.configuration. I am planning to update file
> > > dt-site.xml in
> > > > folder ~/.dt and default properties file available in apex
> > > archetype to
> > > > set jvm options as follows:
> > > > <property>
> > > > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
> > > > <value>-Dlog4j.configuration=log4j.props</value>
> > > > </property>
> > > >
> > > > And I will copy log4j.props file in ~/.dt folder as well as
> > > > apex-archetypes.
> > > >
> > > > Lastly if someone still miss this new log4j properties file or
> jvm
> > > option
> > > > to set -Dlog4j.configuration we will not put log file name in
> event
> > > raised
> > > > by container or operator.
> > > >
> > > > Please provide your feedback on this approach.
> > > >
> > > > -Priyanka
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: Adding new log4j appender to Apex core
Posted by David Yan <da...@datatorrent.com>.
The goal of this log4j appender is to provide a log offset and the fixed
name of the container log file (instead of apex.log becoming apex.log.1 and
then apex.log.2, etc due to rotation) as part of an error STRAM event so
users can easily locate the log entries around the error.
The user can override the appender, but in that case, the engine detects
that and will not include the log location as part of the STRAM event.
David
On Tue, Nov 22, 2016 at 7:10 PM, Priyanka Gugale <pr...@datatorrent.com>
wrote:
> Hi,
>
> Thomas,
> Yes log4j is ultimately owned by user, and they should be able to override
> it. What I am trying to do is provide a default behavior for Apex. In case
> user isn't using any logger of their own we should use this new appender of
> Apex rather than using standard log4j appender as per hadoop config.
>
> Sanjay,
> Archetype is the good place to put this and I will add it there, but many
> time people won't use it. So I wanted to keep it at ~/.dt as well. Is there
> any other default config folder for Apex?
>
> Also I am not relying on anything. If we fail to find config in app jar or
> ~/.dt we are going to skip usage of this new appender.
>
> -Priyanka
>
> On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <sa...@datatorrent.com>
> wrote:
>
> > The only way to “enforce” this new appender is to update the archetypes
> > (apex-app-archetype and apex-conf-archetype under apex-core/ ) to use
> the
> > new ones as default. But there does not seem to be a way to enforce this
> > for anyone not using the archetypes.
> >
> > I agree with not relying on ~/.dt in apex-core.
> >
> > On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
> >
> > The log4j configuration is ultimately owned by the user, so how do
> you
> > want
> > to enforce a custom appender?
> >
> > I don't think that this should rely on anything in ~/.dt either
> >
> > Thomas
> >
> > On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> > priyanka@datatorrent.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am working on APEXCORE-563
> > > <https://issues.apache.org/jira/browse/APEXCORE-563>
> > > As per this Jira we should put log file name in container/operator
> > events.
> > > The problem is current RollingFileAppender keeps renaming files
> from
> > 1 to 2
> > > to ... n as files reach maximum allowed file size. Because of
> > constant
> > > renaming of files we can't put a fixed file name in stram event.
> > >
> > > To overcome this I would like to add a new log4j appender to
> > ApexCore.
> > > There are two ways I can implement this:
> > > 1. Have Daily rolling file appender. The current file will be
> > recognized
> > > based on timestamp in file name. Also to control max file size, we
> > need to
> > > keep rolling files based on size as well.
> > > 2. Have Rolling File Appender but do not rename files. When max
> file
> > size
> > > is reached create new file with next number e.g. crate log file
> > dt.log.2
> > > after dt.log.1 is full. Also to recognize the latest file keep the
> > softlink
> > > named dt.log pointing to current log file.
> > >
> > > I would prefer to implement approach 2. Please provide your
> > > comments/feedback if you feel otherwise.
> > >
> > > Also to use this new appender we need to use our custom
> > log4j.properties
> > > file instead of one present in hadoop conf. For that we need to set
> > jvm
> > > option -Dlog4j.configuration. I am planning to update file
> > dt-site.xml in
> > > folder ~/.dt and default properties file available in apex
> > archetype to
> > > set jvm options as follows:
> > > <property>
> > > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
> > > <value>-Dlog4j.configuration=log4j.props</value>
> > > </property>
> > >
> > > And I will copy log4j.props file in ~/.dt folder as well as
> > > apex-archetypes.
> > >
> > > Lastly if someone still miss this new log4j properties file or jvm
> > option
> > > to set -Dlog4j.configuration we will not put log file name in event
> > raised
> > > by container or operator.
> > >
> > > Please provide your feedback on this approach.
> > >
> > > -Priyanka
> > >
> >
> >
> >
> >
>
Re: Adding new log4j appender to Apex core
Posted by Priyanka Gugale <pr...@datatorrent.com>.
Hi,
Thomas,
Yes log4j is ultimately owned by user, and they should be able to override
it. What I am trying to do is provide a default behavior for Apex. In case
user isn't using any logger of their own we should use this new appender of
Apex rather than using standard log4j appender as per hadoop config.
Sanjay,
Archetype is the good place to put this and I will add it there, but many
time people won't use it. So I wanted to keep it at ~/.dt as well. Is there
any other default config folder for Apex?
Also I am not relying on anything. If we fail to find config in app jar or
~/.dt we are going to skip usage of this new appender.
-Priyanka
On Wed, Nov 23, 2016 at 5:58 AM, Sanjay Pujare <sa...@datatorrent.com>
wrote:
> The only way to “enforce” this new appender is to update the archetypes
> (apex-app-archetype and apex-conf-archetype under apex-core/ ) to use the
> new ones as default. But there does not seem to be a way to enforce this
> for anyone not using the archetypes.
>
> I agree with not relying on ~/.dt in apex-core.
>
> On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
>
> The log4j configuration is ultimately owned by the user, so how do you
> want
> to enforce a custom appender?
>
> I don't think that this should rely on anything in ~/.dt either
>
> Thomas
>
> On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <
> priyanka@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > I am working on APEXCORE-563
> > <https://issues.apache.org/jira/browse/APEXCORE-563>
> > As per this Jira we should put log file name in container/operator
> events.
> > The problem is current RollingFileAppender keeps renaming files from
> 1 to 2
> > to ... n as files reach maximum allowed file size. Because of
> constant
> > renaming of files we can't put a fixed file name in stram event.
> >
> > To overcome this I would like to add a new log4j appender to
> ApexCore.
> > There are two ways I can implement this:
> > 1. Have Daily rolling file appender. The current file will be
> recognized
> > based on timestamp in file name. Also to control max file size, we
> need to
> > keep rolling files based on size as well.
> > 2. Have Rolling File Appender but do not rename files. When max file
> size
> > is reached create new file with next number e.g. crate log file
> dt.log.2
> > after dt.log.1 is full. Also to recognize the latest file keep the
> softlink
> > named dt.log pointing to current log file.
> >
> > I would prefer to implement approach 2. Please provide your
> > comments/feedback if you feel otherwise.
> >
> > Also to use this new appender we need to use our custom
> log4j.properties
> > file instead of one present in hadoop conf. For that we need to set
> jvm
> > option -Dlog4j.configuration. I am planning to update file
> dt-site.xml in
> > folder ~/.dt and default properties file available in apex
> archetype to
> > set jvm options as follows:
> > <property>
> > <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
> > <value>-Dlog4j.configuration=log4j.props</value>
> > </property>
> >
> > And I will copy log4j.props file in ~/.dt folder as well as
> > apex-archetypes.
> >
> > Lastly if someone still miss this new log4j properties file or jvm
> option
> > to set -Dlog4j.configuration we will not put log file name in event
> raised
> > by container or operator.
> >
> > Please provide your feedback on this approach.
> >
> > -Priyanka
> >
>
>
>
>
Re: Adding new log4j appender to Apex core
Posted by Sanjay Pujare <sa...@datatorrent.com>.
The only way to “enforce” this new appender is to update the archetypes (apex-app-archetype and apex-conf-archetype under apex-core/ ) to use the new ones as default. But there does not seem to be a way to enforce this for anyone not using the archetypes.
I agree with not relying on ~/.dt in apex-core.
On 11/22/16, 1:08 PM, "Thomas Weise" <th...@apache.org> wrote:
The log4j configuration is ultimately owned by the user, so how do you want
to enforce a custom appender?
I don't think that this should rely on anything in ~/.dt either
Thomas
On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <pr...@datatorrent.com>
wrote:
> Hi,
>
> I am working on APEXCORE-563
> <https://issues.apache.org/jira/browse/APEXCORE-563>
> As per this Jira we should put log file name in container/operator events.
> The problem is current RollingFileAppender keeps renaming files from 1 to 2
> to ... n as files reach maximum allowed file size. Because of constant
> renaming of files we can't put a fixed file name in stram event.
>
> To overcome this I would like to add a new log4j appender to ApexCore.
> There are two ways I can implement this:
> 1. Have Daily rolling file appender. The current file will be recognized
> based on timestamp in file name. Also to control max file size, we need to
> keep rolling files based on size as well.
> 2. Have Rolling File Appender but do not rename files. When max file size
> is reached create new file with next number e.g. crate log file dt.log.2
> after dt.log.1 is full. Also to recognize the latest file keep the softlink
> named dt.log pointing to current log file.
>
> I would prefer to implement approach 2. Please provide your
> comments/feedback if you feel otherwise.
>
> Also to use this new appender we need to use our custom log4j.properties
> file instead of one present in hadoop conf. For that we need to set jvm
> option -Dlog4j.configuration. I am planning to update file dt-site.xml in
> folder ~/.dt and default properties file available in apex archetype to
> set jvm options as follows:
> <property>
> <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
> <value>-Dlog4j.configuration=log4j.props</value>
> </property>
>
> And I will copy log4j.props file in ~/.dt folder as well as
> apex-archetypes.
>
> Lastly if someone still miss this new log4j properties file or jvm option
> to set -Dlog4j.configuration we will not put log file name in event raised
> by container or operator.
>
> Please provide your feedback on this approach.
>
> -Priyanka
>
Re: Adding new log4j appender to Apex core
Posted by Thomas Weise <th...@apache.org>.
The log4j configuration is ultimately owned by the user, so how do you want
to enforce a custom appender?
I don't think that this should rely on anything in ~/.dt either
Thomas
On Tue, Nov 22, 2016 at 10:00 AM, Priyanka Gugale <pr...@datatorrent.com>
wrote:
> Hi,
>
> I am working on APEXCORE-563
> <https://issues.apache.org/jira/browse/APEXCORE-563>
> As per this Jira we should put log file name in container/operator events.
> The problem is current RollingFileAppender keeps renaming files from 1 to 2
> to ... n as files reach maximum allowed file size. Because of constant
> renaming of files we can't put a fixed file name in stram event.
>
> To overcome this I would like to add a new log4j appender to ApexCore.
> There are two ways I can implement this:
> 1. Have Daily rolling file appender. The current file will be recognized
> based on timestamp in file name. Also to control max file size, we need to
> keep rolling files based on size as well.
> 2. Have Rolling File Appender but do not rename files. When max file size
> is reached create new file with next number e.g. crate log file dt.log.2
> after dt.log.1 is full. Also to recognize the latest file keep the softlink
> named dt.log pointing to current log file.
>
> I would prefer to implement approach 2. Please provide your
> comments/feedback if you feel otherwise.
>
> Also to use this new appender we need to use our custom log4j.properties
> file instead of one present in hadoop conf. For that we need to set jvm
> option -Dlog4j.configuration. I am planning to update file dt-site.xml in
> folder ~/.dt and default properties file available in apex archetype to
> set jvm options as follows:
> <property>
> <name>dt.attr.CONTAINER_JVM_OPTIONS</name>
> <value>-Dlog4j.configuration=log4j.props</value>
> </property>
>
> And I will copy log4j.props file in ~/.dt folder as well as
> apex-archetypes.
>
> Lastly if someone still miss this new log4j properties file or jvm option
> to set -Dlog4j.configuration we will not put log file name in event raised
> by container or operator.
>
> Please provide your feedback on this approach.
>
> -Priyanka
>