Unable to log any data captured from kafka

[Fidea Lidea <li...@gmail.com>]

Hi Team,

I've written a flink job & enabled *slf4j* logging mechanism for it.
Flow of* Flink Job :*  Kafka source => process datastream
elements(Transformations) => kafka sink.

It stops logging while processing datastream. I want to log all data
captured from kafka either in a log file or on the stdout console.
I've tried a few approaches but am unable to log data.

*Approaches tried : *

1. dataStream.print() or dataStream.print().toString()

2.Iterator<DTO> myOutput = DataStreamUtils.collect(dataStream);

while (myOutput.hasNext()) {
    log.info("myOutput" + myOutput.next());
    LOG.debug("LOG" + myOutput.next());
}
3.IterativeStream<DTO> iteration = dataStream.iterate();

DataStream<DTO> ge=iteration.map(new MapFunction<DTO, DTO>() {
    @Override
    public DTO map(DTO obj) throws Exception {
        log.info("id  "+obj.getID());
        return obj;
    }


});

4. Store data in a static map & log it before kafka sink etc.

Thanks
Nida

Re:Unable to log any data captured from kafka

[Xuyang <xy...@163.com>]

Hi, Nida.




I'd like to confirm whether there would be any log output if it's executed directly in the IDE. 




If there are logs in the IDE but not when running by submission, you could check if the log configuration files in the TM logs are normal. 

If there are no logs in the IDE either, I've encountered this before, and it was an issue with jar package conflicts.




--

    Best！
    Xuyang




At 2024-05-15 15:20:45, "Fidea Lidea" <li...@gmail.com> wrote:

Hi Team,

I've written a flink job & enabled slf4j logging mechanism for it.
Flow ofFlink Job :  Kafka source => process datastream elements(Transformations) => kafka sink.

It stops logging while processing datastream. I want to log all data captured from kafka either in a log file or on the stdout console.
I've tried a few approaches but am unable to log data.  

Approaches tried : 

1. dataStream.print() or dataStream.print().toString()

2.Iterator<DTO> myOutput = DataStreamUtils.collect(dataStream);
while (myOutput.hasNext()) {
log.info("myOutput" + myOutput.next());
LOG.debug("LOG" + myOutput.next());
}
3.IterativeStream<DTO> iteration = dataStream.iterate();

DataStream<DTO> ge=iteration.map(new MapFunction<DTO, DTO>() {
@Override
public DTO map(DTO obj) throws Exception {
log.info("id  "+obj.getID());
return obj;
    }


});
4. Store data in a static map & log it before kafka sink etc.

Thanks
Nida