[jira] [Commented] (CASSANALYTICS-177) Add instanceId query parameter to fix HTTP 421 errors when Sidecar is behind an AWS Load Balancer

["Bianca Stanciu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANALYTICS-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110543#comment-18110543 ] 

Bianca Stanciu commented on CASSANALYTICS-177:
----------------------------------------------

[https://github.com/apache/cassandra-analytics/pull/222]   is ready for another look.

It attaches an optional instanceId query param to outbound Sidecar requests, so Sidecar
can resolve the intended instance even when the request comes through the NLB and the Host
header ends up being the NLB address instead of the node's (that's what causes the 421).
The routing side doesn't need any Sidecar change: AbstractHandler already prefers this param
over the Host header. What did need a Sidecar change is how the client learns which id to
send per replica. That's CASSSIDECAR-495, which now reports each replica's instance id in
the ring/token-range response. There's also a static fallback (host=id on a contact point,
or the job-level sidecar.instance.id) for simpler setups and older Sidecar versions.

Per-instance ids take precedence over the job-level one, so a multi-node fan-out routes
correctly. And if a single job-level id would end up stamped onto multiple instances (more
than one instance in the ring, at least one without its own id), it fails fast instead of
misrouting. The new field is nullable and backward compatible both ways, and there are unit
tests for all of this.

Scope here is the bulk writer path. One thing I'd like your take on: the read path isn't
covered. CassandraDataLayer and SidecarTableSizeProvider still build their Sidecar instances
without an id, so a read behind the NLB would hit the same 421. Does it make sense to handle
that here, or would you rather a separate ticket for it?

 

cc: [~skoppu] 

 

> Add instanceId query parameter to fix HTTP 421 errors when Sidecar is behind an AWS Load Balancer
> -------------------------------------------------------------------------------------------------
>
>                 Key: CASSANALYTICS-177
>                 URL: https://issues.apache.org/jira/browse/CASSANALYTICS-177
>             Project: Apache Cassandra Analytics
>          Issue Type: Improvement
>          Components: Writer
>            Reporter: Bianca Stanciu
>            Assignee: Bianca Stanciu
>            Priority: High
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> +_*Problem:*_+
> When Sidecar sits behind an AWS NLB, every HTTP request from the analytics client returns *421 Misdirected Request.*
> The NLB forwards TCP without modifying HTTP headers, so the Host header arriving at Sidecar contains the NLB's IP, not the Sidecar's own address. Sidecar uses Host header to identify which Cassandra instance the request is for. 
> _*Solution:*_ 
> Add an optional instanceId query parameter appended to every outbound Sidecar request:
> {code:java}
> POST /api/v1/restore-job/slices?instanceId=1
> GET /api/v1/cassandra/timeSkew?instanceId=1{code}
> Sidecar already checks +_*?instanceId*_+ before the Host header, bypassing the NLB IP lookup entirely. No Sidecar-side changes needed. 
> Enabled with one Spark conf flag:
> {code:java}
> --conf "spark.cassandra_analytics.sidecar.instance.id=1" {code}
> Default is null, no parameter added, zero behavior change for the existing flow.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org