You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Puneet Kinra <pu...@customercentria.com> on 2018/03/20 09:36:44 UTC
Help Required for Log4J
Hi
I have a use case in which i want to log bad records in the log file. I
have configured the log4j
property file is getting generated as well but it also going to flink logs
as well i want to detach
it from flink logs want to write to log file.
.Here is configuration
*(Note :AMSSource is the custom written adaptor here)*
# This affects logging for both user code and Flink
log4j.rootLogger=INFO, file
log4j.logger.amssource=DEBUG, amssourceAppender
# Uncomment this if you want to _only_ change Flink's logging
#log4j.logger.org.apache.flink=INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to
manually
# change the log levels here.
log4j.logger.akka=INFO
log4j.logger.org.apache.kafka=INFO
log4j.logger.org.apache.hadoop=INFO
log4j.logger.org.apache.zookeeper=INFO
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=D:\\logs\\flink-log
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,
file
#BonusPointAppender
log4j.appender.bonuspointAppender=org.apache.log4j.RollingFileAppender
log4j.appender.bonuspointAppender.MaxFileSize=1024MB
log4j.appender.bonuspointAppender.MaxBackupIndex=10
log4j.appender.bonuspointAppender.Append=true
log4j.appender.bonuspointAppender.File=D:\\logs\\flink-bpuser-bonus.logs
#log4j.appender.bonuspointAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.bonuspointAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.bonuspointAppender.layout.ConversionPattern=%d [%t] %-5p (%C
%M:%L) %x - %m%n
#AMSSourceAppender
log4j.appender.amssourceAppender=org.apache.log4j.RollingFileAppender
log4j.appender.amssourceAppender.MaxFileSize=1024MB
log4j.appender.amssourceAppender.MaxBackupIndex=10
log4j.appender.amssourceAppender.Append=true
log4j.appender.amssourceAppender.File=D:\\logs\\flink-bpuser-bonus-
amssource.logs
#log4j.appender.amssourceAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.amssourceAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.amssourceAppender.layout.ConversionPattern=%d [%t] %-5p (%C
%M:%L) %x - %m%n
--
*Cheers *
*Puneet Kinra*
*Mobile:+918800167808 | Skype : puneet.kinra@customercentria.com
<pu...@customercentria.com>*
*e-mail :puneet.kinra@customercentria.com
<pu...@customercentria.com>*
Re: Help Required for Log4J
Posted by Puneet Kinra <pu...@customercentria.com>.
Hi
Fabin thanks for reply I fixed the issue that i was facing.
On Tue, Mar 20, 2018 at 7:31 PM, Fabian Hueske <fh...@gmail.com> wrote:
> Hi,
>
> TBH, I don't have much experience with logging, but you might want to
> consider using Side Outputs [1] to route invalid records into a separate
> stream.
> The stream can then separately handled, be written to files or Kafka or
> wherever.
>
> Best,
> Fabian
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.4/dev/stream/side_output.html
>
> 2018-03-20 10:36 GMT+01:00 Puneet Kinra <pu...@customercentria.com>
> :
>
>> Hi
>>
>> I have a use case in which i want to log bad records in the log file. I
>> have configured the log4j
>> property file is getting generated as well but it also going to flink
>> logs as well i want to detach
>> it from flink logs want to write to log file.
>>
>> .Here is configuration
>> *(Note :AMSSource is the custom written adaptor here)*
>>
>> # This affects logging for both user code and Flink
>> log4j.rootLogger=INFO, file
>> log4j.logger.amssource=DEBUG, amssourceAppender
>>
>> # Uncomment this if you want to _only_ change Flink's logging
>> #log4j.logger.org.apache.flink=INFO
>>
>> # The following lines keep the log level of common libraries/connectors on
>> # log level INFO. The root logger does not override this. You have to
>> manually
>> # change the log levels here.
>> log4j.logger.akka=INFO
>> log4j.logger.org.apache.kafka=INFO
>> log4j.logger.org.apache.hadoop=INFO
>> log4j.logger.org.apache.zookeeper=INFO
>>
>> # Log all infos in the given file
>> log4j.appender.file=org.apache.log4j.FileAppender
>> log4j.appender.file.file=D:\\logs\\flink-log
>> log4j.appender.file.append=false
>> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>> log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
>> %-5p %-60c %x - %m%n
>>
>> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
>> log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.ch
>> annel.DefaultChannelPipeline=ERROR, file
>>
>>
>> #BonusPointAppender
>> log4j.appender.bonuspointAppender=org.apache.log4j.RollingFileAppender
>> log4j.appender.bonuspointAppender.MaxFileSize=1024MB
>> log4j.appender.bonuspointAppender.MaxBackupIndex=10
>> log4j.appender.bonuspointAppender.Append=true
>> log4j.appender.bonuspointAppender.File=D:\\logs\\flink-bpuser-bonus.logs
>> #log4j.appender.bonuspointAppender.DatePattern='.'yyyy-MM-dd
>> log4j.appender.bonuspointAppender.layout=org.apache.log4j.PatternLayout
>> log4j.appender.bonuspointAppender.layout.ConversionPattern=%d [%t] %-5p
>> (%C %M:%L) %x - %m%n
>>
>> #AMSSourceAppender
>> log4j.appender.amssourceAppender=org.apache.log4j.RollingFileAppender
>> log4j.appender.amssourceAppender.MaxFileSize=1024MB
>> log4j.appender.amssourceAppender.MaxBackupIndex=10
>> log4j.appender.amssourceAppender.Append=true
>> log4j.appender.amssourceAppender.File=D:\\logs\\flink-bpuser
>> -bonus-amssource.logs
>> #log4j.appender.amssourceAppender.DatePattern='.'yyyy-MM-dd
>> log4j.appender.amssourceAppender.layout=org.apache.log4j.PatternLayout
>> log4j.appender.amssourceAppender.layout.ConversionPattern=%d [%t] %-5p
>> (%C %M:%L) %x - %m%n
>>
>>
>>
>>
>> --
>> *Cheers *
>>
>> *Puneet Kinra*
>>
>> *Mobile:+918800167808 <+91%2088001%2067808> | Skype :
>> puneet.kinra@customercentria.com <pu...@customercentria.com>*
>>
>> *e-mail :puneet.kinra@customercentria.com
>> <pu...@customercentria.com>*
>>
>>
>>
>
--
*Cheers *
*Puneet Kinra*
*Mobile:+918800167808 | Skype : puneet.kinra@customercentria.com
<pu...@customercentria.com>*
*e-mail :puneet.kinra@customercentria.com
<pu...@customercentria.com>*
Re: Help Required for Log4J
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
TBH, I don't have much experience with logging, but you might want to
consider using Side Outputs [1] to route invalid records into a separate
stream.
The stream can then separately handled, be written to files or Kafka or
wherever.
Best,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/side_output.html
2018-03-20 10:36 GMT+01:00 Puneet Kinra <pu...@customercentria.com>:
> Hi
>
> I have a use case in which i want to log bad records in the log file. I
> have configured the log4j
> property file is getting generated as well but it also going to flink logs
> as well i want to detach
> it from flink logs want to write to log file.
>
> .Here is configuration
> *(Note :AMSSource is the custom written adaptor here)*
>
> # This affects logging for both user code and Flink
> log4j.rootLogger=INFO, file
> log4j.logger.amssource=DEBUG, amssourceAppender
>
> # Uncomment this if you want to _only_ change Flink's logging
> #log4j.logger.org.apache.flink=INFO
>
> # The following lines keep the log level of common libraries/connectors on
> # log level INFO. The root logger does not override this. You have to
> manually
> # change the log levels here.
> log4j.logger.akka=INFO
> log4j.logger.org.apache.kafka=INFO
> log4j.logger.org.apache.hadoop=INFO
> log4j.logger.org.apache.zookeeper=INFO
>
> # Log all infos in the given file
> log4j.appender.file=org.apache.log4j.FileAppender
> log4j.appender.file.file=D:\\logs\\flink-log
> log4j.appender.file.append=false
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
> log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.ch
> annel.DefaultChannelPipeline=ERROR, file
>
>
> #BonusPointAppender
> log4j.appender.bonuspointAppender=org.apache.log4j.RollingFileAppender
> log4j.appender.bonuspointAppender.MaxFileSize=1024MB
> log4j.appender.bonuspointAppender.MaxBackupIndex=10
> log4j.appender.bonuspointAppender.Append=true
> log4j.appender.bonuspointAppender.File=D:\\logs\\flink-bpuser-bonus.logs
> #log4j.appender.bonuspointAppender.DatePattern='.'yyyy-MM-dd
> log4j.appender.bonuspointAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.bonuspointAppender.layout.ConversionPattern=%d [%t] %-5p
> (%C %M:%L) %x - %m%n
>
> #AMSSourceAppender
> log4j.appender.amssourceAppender=org.apache.log4j.RollingFileAppender
> log4j.appender.amssourceAppender.MaxFileSize=1024MB
> log4j.appender.amssourceAppender.MaxBackupIndex=10
> log4j.appender.amssourceAppender.Append=true
> log4j.appender.amssourceAppender.File=D:\\logs\\flink-
> bpuser-bonus-amssource.logs
> #log4j.appender.amssourceAppender.DatePattern='.'yyyy-MM-dd
> log4j.appender.amssourceAppender.layout=org.apache.log4j.PatternLayout
> log4j.appender.amssourceAppender.layout.ConversionPattern=%d [%t] %-5p
> (%C %M:%L) %x - %m%n
>
>
>
>
> --
> *Cheers *
>
> *Puneet Kinra*
>
> *Mobile:+918800167808 <+91%2088001%2067808> | Skype :
> puneet.kinra@customercentria.com <pu...@customercentria.com>*
>
> *e-mail :puneet.kinra@customercentria.com
> <pu...@customercentria.com>*
>
>
>