[GitHub] [pulsar] Remco2376 commented on issue #9186: record.getPublishTime return empty

[GitBox <gi...@apache.org>]

Remco2376 commented on issue #9186:
URL: https://github.com/apache/pulsar/issues/9186#issuecomment-759499499


   I can confirm that Pulsar sink connectors occasionally receive records that do not contain eventTime and/or message. This behavior seems random. These Java calls fail randomly:
   
   long eventTime = record.getEventTime().orElseThrow(() -> new RuntimeException("EventTime does not exist in record"));
   Message<GenericRecord> message = record.getMessage().orElseThrow(() -> new RuntimeException("Message does not exist in record"));
   
   All records to actually contain an eventTime and message.
   
   This happened with Pulsar 2.6.1. Recently upgraded to 2.7.0 and haven't seen the same issue so far. But I've tested only for a very limited time. This issue was observed in Pulsar/Docker (https://pulsar.apache.org/docs/en/standalone-docker/).
   
   I'm not sure, but I got the impression that this issue could be related to high memory usage (https://github.com/streamnative/pulsar-io-cloud-storage/blob/master/src/main/java/org/apache/pulsar/io/jcloud/util/AvroRecordUtil.java contains a memory leak), so I'm currently logging memory usage to confirm that:
   
   long totalMemory = Runtime.getRuntime().totalMemory();
   long freeMemory = Runtime.getRuntime().freeMemory();
   long memoryUsage = totalMemory - freeMemory;
   
   System.out.println(totalMemory + " " + freeMemory + " " + memoryUsage);


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org