Flink BatchJob Error - Sort Buffer, Record too large.

[SURAJ KADEL <su...@gmail.com>]

Hi Team,
I am currently running batch jobs on flink 1.19.2 and the expected
throughput for a certain step is above 1TB.
What I have observed is that everytime my job reaches close to 30%, it
fails with the below error after a few retries. I am currently running this
on kubernetes and using 6 CPU and 48GB Memory resources for TaskManager and
3 CPU and 32GB Memory for JobManager.

Caused by: java.io.IOException: Cannot write record to fresh sort buffer.
Record too large.
at org.apache.flink.runtime.operators.chaining.
SynchronousChainedCombineDriver.collect(SynchronousChainedCombineDriver
.java:190)
at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect
(CountingCollector.java:35)
at org.apache.flink.runtime.operators.chaining.ChainedMapDriver.collect(
ChainedMapDriver.java:78)
at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect
(CountingCollector.java:35)
at org.apache.flink.api.java.operators.translation.
PlanFilterOperator$FlatMapFilter.flatMap(PlanFilterOperator.java:58)
at org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.java:
113)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:516)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:359)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(
Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.base/java.lang.Thread.run(Unknown Source)
I have two questions here,
1.How can I handle this issue of sort buffer, Record too large?
2.What is the recommended way to handle large records in flink Batch Jobs?

Please help me out with this.

Regards,
Suraj