You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kant kodali <ka...@gmail.com> on 2019/03/23 02:02:05 UTC
How to control batch size while reading from hdfs files?
Hi All,
What determines the batch size while reading from a file from HDFS?
I am trying to read files from HDFS and ingest into Kafka using Spark
Structured Streaming 2.3.1. I get an error sayiKafkafka batch size is too
big and that I need to increase max.request.size. Sure I can increase it
but I would like to know what other parameters I can change such that I
don't have to change the default max.request.size?
The default max.request.size of a Kafka producer docs says it is set to
1MB.
And each file I have in HDFS is < 12MB.
Any suggestions will be great.
Thanks!