[jira] [Created] (CAMEL-25021) Kafka consumer silently stalls indefinitely when record exceeds max.partition.fetch.bytes

["Rakshit Agarwal (Jira)" <ji...@apache.org>]

Rakshit Agarwal created CAMEL-25021:
---------------------------------------

             Summary: Kafka consumer silently stalls indefinitely when record exceeds max.partition.fetch.bytes
                 Key: CAMEL-25021
                 URL: https://issues.apache.org/jira/browse/CAMEL-25021
             Project: Camel
          Issue Type: Bug
          Components: camel-kafka
    Affects Versions: 4.14.5
            Reporter: Rakshit Agarwal
             Fix For: 4.14.5


The camel-kafka consumer enters an infinite silent loop when a Kafka topic contains a record larger than the consumer's max.partition.fetch.bytes (default 1MB). The consumer calls poll() repeatedly, receives empty ConsumerRecords, processes zero records, and loops back — with no error, no log, and no progress. The consumer appears alive but is permanently stalled.

*Root Cause*

In KafkaFetchRecords.java, the main poll loop (line 376):

  \{code:java}
  while (isKafkaConsumerRunnableAndNotStopped() && isConnected() && pollExceptionStrategy.canContinue()) {
      ConsumerRecords<Object, Object> allRecords = consumer.poll(pollDuration);
      ProcessingResult result = recordProcessorFacade.processPolledRecords(allRecords);
      updateTaskState();
      if (result != null && result.isBreakOnErrorHit() && !this.state.get().equals(State.PAUSED)) {
          setReconnect(true);
          setConnected(false);
      }
  }
  \{code}

When the Kafka broker encounters a record larger than max.partition.fetch.bytes, it returns an empty fetch response — this is correct Kafka protocol behavior, the broker cannot fit the record into the fetch buffer. The Camel consumer receives empty ConsumerRecords, processPolledRecords processes zero records without error, and the while loop continues.

There is no detection for the "stuck on same offset with repeated empty polls" condition. The pollExceptionStrategy is never triggered because no exception occurs — the poll succeeds, it just returns nothing. No code path in KafkaFetchRecords tracks whether the consumer offset is advancing or stalled.

The KafkaConfiguration class exposes maxPartitionFetchBytes as a configurable property, but does not set a default — inheriting the Kafka client's default of 1,048,576 bytes (1MB). This means any record >1MB published to the topic by any producer becomes a permanent poison pill that silently stalls the consumer.

On our side, we have added a 5MB MessageSizeLimitProcessor at the application level to reject oversized messages after they are consumed. However, this processor is never reached for messages >1MB because the consumer stalls before fetching them — making the application-level size check unreachable.

The only current workaround is to increase maxPartitionFetchBytes to a value larger than the largest possible message (e.g., 50MB). This trades the silent stall for excessive memory
consumption — the entire oversized record is fetched into consumer heap, deserialized into a Java String, and only then rejected by application-level checks downstream. A 50MB message consumes ~100MB of heap (byte[] + String) before rejection.

*Steps to Reproduce*

  # Create a Kafka topic with max.message.bytes set to allow large messages (e.g., 10MB)
  # Produce a message larger than 1MB (e.g., 6MB) to the topic
  # Produce several normal-sized messages after the oversized one
  # Start a camel-kafka consumer route with default configuration (no maxPartitionFetchBytes override)
  # Observe: the consumer processes messages up to the offset before the oversized record, then stalls indefinitely
  # Check consumer group: CURRENT-OFFSET stops advancing. Check logs: no error, no warning at any level

*Expected Behavior*

The consumer should detect that it is stuck at the same offset receiving consecutive empty polls and surface an error — either by throwing an exception routable to the configured errorHandler, or by logging a WARN that clearly identifies the stuck offset, partition, and max.partition.fetch.bytes value. The consumer should not silently loop forever with no indication of the problem.

*Actual Behavior*

The consumer loops indefinitely with no error, no warning, and no progress. The poll loop has no tracking of whether the offset is advancing. The pollExceptionStrategy is never invoked because no exception occurs. TRACE-level "Polling..." messages continue but provide no indication of the stall.

*Impact / Severity*

  * Any single oversized record in any partition permanently stalls the consumer for that partition with no log output
  * Restarting the consumer does not recover — it resumes from the same stuck offset and stalls again
  * The consumer thread remains alive and the consumer group shows the member as active — standard monitoring (thread health, consumer group lag alerts) may not detect the stall because lag appears constant rather than growing
  * No exception is surfaced to the Camel route, error handler, or application



--
This message was sent by Atlassian Jira
(v8.20.10#820010)