You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Jason Gerlowski (Jira)" <ji...@apache.org> on 2023/04/07 13:28:00 UTC
[jira] [Comment Edited] (SOLR-16720) PKI should decorate outgoing requests at "sending", not "enqueueing" time
[ https://issues.apache.org/jira/browse/SOLR-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709679#comment-17709679 ]
Jason Gerlowski edited comment on SOLR-16720 at 4/7/23 1:27 PM:
----------------------------------------------------------------
Anything is possible. But I'm not convinced necessarily - I'm not seeing the particular logging I'd expect if the PKI header was missing or set with incomplete data. Though I'll definitely look closer.
JWTAuthPluginIntegrationTest has had terrible levels of flakiness going back all the way to the start of 2022. And while it's a shame to even talk this way, the failures this week seem about "normal" for that test.
EDIT: Deleted a screenshot because it displayed terribly in JIRA, but see the fucit link [here|http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.mockOAuth2Server] for the historical test failure trend.
I'll take a closer look at the logs from some of the failures this afternoon though. And I'm happy to rollback out of an abundance of caution if you think it's warranted, or if you're just curious how the builds might look without this change?
was (Author: gerlowskija):
Anything is possible. But I'm not convinced necessarily - I'm not seeing the particular logging I'd expect if the PKI header was missing or set with incomplete data. Though I'll definitely look closer.
JWTAuthPluginIntegrationTest has had terrible levels of flakiness going back all the way to the start of 2022. And while it's a shame to even talk this way, the failures this week seem about "normal" for that test.
!Screen Shot 2023-04-07 at 9.16.30 AM.png!
(Fucit link [here|http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.mockOAuth2Server] fwiw)
I'll take a closer look at the logs from some of the failures this afternoon though. And I'm happy to rollback out of an abundance of caution if you think it's warranted, or if you're just curious how the builds might look without this change?
> PKI should decorate outgoing requests at "sending", not "enqueueing" time
> -------------------------------------------------------------------------
>
> Key: SOLR-16720
> URL: https://issues.apache.org/jira/browse/SOLR-16720
> Project: Solr
> Issue Type: Improvement
> Components: Authentication
> Affects Versions: 9.2
> Reporter: Jason Gerlowski
> Priority: Minor
> Attachments: SOLR-16720-reproduce.patch, Screen Shot 2023-04-07 at 9.16.30 AM.png, reproduce.sh
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, PKIAuthenticationPlugin decorates intra-node requests using an 'onQueue' lifecycle hook, which is triggered when the request is enqueued for processing by the (asynchronous) Jetty http client.
> This works great on many systems. However on heavily loaded clusters the time between Jetty "queueing" the request and it actually being sent out can be non-negligible. If this gap becomes wide enough, the TTL encoded into the PKI auth header might have substantially or fully expired by the time the receiving node gets the request.
> We should experiment with moving PKI header decoration to the 'onBegin' hook instead, which fires much closer to the actual request-send time on heavily loaded servers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org