You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chitra Elangovan <ch...@gmail.com> on 2026/01/22 00:56:08 UTC
Increasing/Exposing requestHeaderSize for Kafka Connect REST API
Hello Kafka Community,
I am reaching out to discuss a limitation regarding the Kafka Connect REST
API's embedded Jetty server (we are using kafka 3.9.0 version).
Currently, the default header size limit is 8KB. In our environment, we are
exceeding this limit due to our business needs. When this limit is
exceeded, Jetty returns a 431 Request Header Fields Too Large error.
I've noted that while listeners and a few other properties are configurable
via the Worker properties, there doesn't seem to be a straightforward way
to increase the requestHeaderSize for the internal HttpConfiguration.
Questions for the group:
- Is there an existing JVM system property or undocumented config that can
override this in the current version?
- If not, would the community be open to a KIP (Kafka Improvement Proposal)
to expose this as a worker configuration (e.g.,
rest.request.header.size.default)?
- Are there any known workarounds for Kafka Connect distributed workers
without using an external reverse proxy to strip headers?Looking forward to
your insights.
Best regards,
Chitra Elangovan
Re: Increasing/Exposing requestHeaderSize for Kafka Connect REST API
Posted by Yeikel Santana <em...@yeikel.com>.
I was curious about this and did a quick search. It appears that this approach is generally considered an anti pattern [1].
Please do not take this as an authoritative opinion as that's my only source. I am simply trying to understand why such a large header is required and if the problem could be solved using the request body or any other way like a proxy.
Jetty does have a rationale for these limits, as they are based on the HTTP specification.
On that note, something you should consider is that If you have other systems in between your client and Kafka Connect (such as load balancers, and friends), you may face the same issue there as well
In any case, if we decide to take the KIP approach to enable this, I assume we will need to provide a justification that goes beyond citing "business requirements" to explain the effort involved in building it and the overall benefits it'll bring to the framework
Thanks!
[1] https://github.com/jetty/jetty.project/issues/12071
From: Chris Egerton <fe...@gmail.com>
To: "Users"<us...@kafka.apache.org>
Date: Thu, 22 Jan 2026 21:49:25 -0500
Subject: Re: Increasing/Exposing requestHeaderSize for Kafka Connect REST API
Ah, I see! Sorry about that, been a sec since I've gotten my hands dirty
with the server logic for Connect.
I can't think of any other workarounds, unfortunately. I'd be in favor of a
small KIP to plug the gap though!
On Thu, Jan 22, 2026, 15:19 Chitra Elangovan < mailto:chitrae92@gmail.com > wrote:
> Hi Chris,
>
> We looked at the ConnectRestExtension.html
> <
> https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html
> >,
> but since the 8KB limit is enforced at the Jetty Server level, the
> extension isn't invoked until after the 431 error is already triggered. Are
> there any other workarounds that are currently available?
>
> Thanks,
> Chitra Elangovan
>
>
> On Wed, Jan 21, 2026 at 5:23 PM Chris Egerton < mailto:fearthecellos@gmail.com >
> wrote:
>
> > Hi Chitra,
> >
> > Unsure about whether this will help with your specific requirements or
> not
> > but have you looked into the REST extension API? Linking to the 3.9.x
> docs
> > since that's the version you're running but AFAIK the feature hasn't
> > changed much or even at all in a few years:
> >
> >
> https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html
> >
> > If this doesn't help, I think a KIP could make sense.
> >
> > Cheers,
> >
> > Chris
> >
> >
> > On Wed, Jan 21, 2026, 19:57 Chitra Elangovan < mailto:chitrae92@gmail.com >
> wrote:
> >
> > > Hello Kafka Community,
> > >
> > > I am reaching out to discuss a limitation regarding the Kafka Connect
> > REST
> > > API's embedded Jetty server (we are using kafka 3.9.0 version).
> > >
> > > Currently, the default header size limit is 8KB. In our environment, we
> > are
> > > exceeding this limit due to our business needs. When this limit is
> > > exceeded, Jetty returns a 431 Request Header Fields Too Large error.
> > >
> > > I've noted that while listeners and a few other properties are
> > configurable
> > > via the Worker properties, there doesn't seem to be a straightforward
> way
> > > to increase the requestHeaderSize for the internal HttpConfiguration.
> > >
> > > Questions for the group:
> > >
> > > - Is there an existing JVM system property or undocumented config that
> > can
> > > override this in the current version?
> > >
> > > - If not, would the community be open to a KIP (Kafka Improvement
> > Proposal)
> > > to expose this as a worker configuration (e.g.,
> > > rest.request.header.size.default)?
> > >
> > > - Are there any known workarounds for Kafka Connect distributed workers
> > > without using an external reverse proxy to strip headers?Looking
> forward
> > to
> > > your insights.
> > >
> > > Best regards,
> > > Chitra Elangovan
> > >
> >
>
Re: Increasing/Exposing requestHeaderSize for Kafka Connect REST API
Posted by Chris Egerton <fe...@gmail.com>.
Ah, I see! Sorry about that, been a sec since I've gotten my hands dirty
with the server logic for Connect.
I can't think of any other workarounds, unfortunately. I'd be in favor of a
small KIP to plug the gap though!
On Thu, Jan 22, 2026, 15:19 Chitra Elangovan <ch...@gmail.com> wrote:
> Hi Chris,
>
> We looked at the ConnectRestExtension.html
> <
> https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html
> >,
> but since the 8KB limit is enforced at the Jetty Server level, the
> extension isn't invoked until after the 431 error is already triggered. Are
> there any other workarounds that are currently available?
>
> Thanks,
> Chitra Elangovan
>
>
> On Wed, Jan 21, 2026 at 5:23 PM Chris Egerton <fe...@gmail.com>
> wrote:
>
> > Hi Chitra,
> >
> > Unsure about whether this will help with your specific requirements or
> not
> > but have you looked into the REST extension API? Linking to the 3.9.x
> docs
> > since that's the version you're running but AFAIK the feature hasn't
> > changed much or even at all in a few years:
> >
> >
> https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html
> >
> > If this doesn't help, I think a KIP could make sense.
> >
> > Cheers,
> >
> > Chris
> >
> >
> > On Wed, Jan 21, 2026, 19:57 Chitra Elangovan <ch...@gmail.com>
> wrote:
> >
> > > Hello Kafka Community,
> > >
> > > I am reaching out to discuss a limitation regarding the Kafka Connect
> > REST
> > > API's embedded Jetty server (we are using kafka 3.9.0 version).
> > >
> > > Currently, the default header size limit is 8KB. In our environment, we
> > are
> > > exceeding this limit due to our business needs. When this limit is
> > > exceeded, Jetty returns a 431 Request Header Fields Too Large error.
> > >
> > > I've noted that while listeners and a few other properties are
> > configurable
> > > via the Worker properties, there doesn't seem to be a straightforward
> way
> > > to increase the requestHeaderSize for the internal HttpConfiguration.
> > >
> > > Questions for the group:
> > >
> > > - Is there an existing JVM system property or undocumented config that
> > can
> > > override this in the current version?
> > >
> > > - If not, would the community be open to a KIP (Kafka Improvement
> > Proposal)
> > > to expose this as a worker configuration (e.g.,
> > > rest.request.header.size.default)?
> > >
> > > - Are there any known workarounds for Kafka Connect distributed workers
> > > without using an external reverse proxy to strip headers?Looking
> forward
> > to
> > > your insights.
> > >
> > > Best regards,
> > > Chitra Elangovan
> > >
> >
>
Re: Increasing/Exposing requestHeaderSize for Kafka Connect REST API
Posted by Chitra Elangovan <ch...@gmail.com>.
Hi Chris,
We looked at the ConnectRestExtension.html
<https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html>,
but since the 8KB limit is enforced at the Jetty Server level, the
extension isn't invoked until after the 431 error is already triggered. Are
there any other workarounds that are currently available?
Thanks,
Chitra Elangovan
On Wed, Jan 21, 2026 at 5:23 PM Chris Egerton <fe...@gmail.com>
wrote:
> Hi Chitra,
>
> Unsure about whether this will help with your specific requirements or not
> but have you looked into the REST extension API? Linking to the 3.9.x docs
> since that's the version you're running but AFAIK the feature hasn't
> changed much or even at all in a few years:
>
> https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html
>
> If this doesn't help, I think a KIP could make sense.
>
> Cheers,
>
> Chris
>
>
> On Wed, Jan 21, 2026, 19:57 Chitra Elangovan <ch...@gmail.com> wrote:
>
> > Hello Kafka Community,
> >
> > I am reaching out to discuss a limitation regarding the Kafka Connect
> REST
> > API's embedded Jetty server (we are using kafka 3.9.0 version).
> >
> > Currently, the default header size limit is 8KB. In our environment, we
> are
> > exceeding this limit due to our business needs. When this limit is
> > exceeded, Jetty returns a 431 Request Header Fields Too Large error.
> >
> > I've noted that while listeners and a few other properties are
> configurable
> > via the Worker properties, there doesn't seem to be a straightforward way
> > to increase the requestHeaderSize for the internal HttpConfiguration.
> >
> > Questions for the group:
> >
> > - Is there an existing JVM system property or undocumented config that
> can
> > override this in the current version?
> >
> > - If not, would the community be open to a KIP (Kafka Improvement
> Proposal)
> > to expose this as a worker configuration (e.g.,
> > rest.request.header.size.default)?
> >
> > - Are there any known workarounds for Kafka Connect distributed workers
> > without using an external reverse proxy to strip headers?Looking forward
> to
> > your insights.
> >
> > Best regards,
> > Chitra Elangovan
> >
>
Re: Increasing/Exposing requestHeaderSize for Kafka Connect REST API
Posted by Chris Egerton <fe...@gmail.com>.
Hi Chitra,
Unsure about whether this will help with your specific requirements or not
but have you looked into the REST extension API? Linking to the 3.9.x docs
since that's the version you're running but AFAIK the feature hasn't
changed much or even at all in a few years:
https://kafka.apache.org/39/javadoc/org/apache/kafka/connect/rest/ConnectRestExtension.html
If this doesn't help, I think a KIP could make sense.
Cheers,
Chris
On Wed, Jan 21, 2026, 19:57 Chitra Elangovan <ch...@gmail.com> wrote:
> Hello Kafka Community,
>
> I am reaching out to discuss a limitation regarding the Kafka Connect REST
> API's embedded Jetty server (we are using kafka 3.9.0 version).
>
> Currently, the default header size limit is 8KB. In our environment, we are
> exceeding this limit due to our business needs. When this limit is
> exceeded, Jetty returns a 431 Request Header Fields Too Large error.
>
> I've noted that while listeners and a few other properties are configurable
> via the Worker properties, there doesn't seem to be a straightforward way
> to increase the requestHeaderSize for the internal HttpConfiguration.
>
> Questions for the group:
>
> - Is there an existing JVM system property or undocumented config that can
> override this in the current version?
>
> - If not, would the community be open to a KIP (Kafka Improvement Proposal)
> to expose this as a worker configuration (e.g.,
> rest.request.header.size.default)?
>
> - Are there any known workarounds for Kafka Connect distributed workers
> without using an external reverse proxy to strip headers?Looking forward to
> your insights.
>
> Best regards,
> Chitra Elangovan
>