You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2026/08/04 23:19:09 UTC

[kudu-CR] [util] fix flakiness in MiniPrometheusTest.StartStopNoTargets

Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24658


Change subject: [util] fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................

[util] fix flakiness in MiniPrometheusTest.StartStopNoTargets

Before this fix, one of my RHEL9.6 x86_64 build/test machines would
regularly report a failure in the MiniPrometheusTest.StartStopNoTargets
scenario when running mini_prometheus-test.  Reports looked like below:

  src/kudu/util/mini_prometheus-test.cc:206: Failure
  Failed
  Bad status: Remote error: HTTP 503

This changelist address the issue.

I also tried a more generic approach, trying to rely on the documented
'/-/ready' URL [1] by running the readiness check right before returning
from the MiniPrometheus::Start() method.  However, it seems getting
HTTP 200 for GET/HEAD requests at the '/-/ready' URL doesn't guarantee
that Scrape/Target Manager initialization is complete, so the readiness
check is useless in this context.

[1] https://prometheus.io/docs/prometheus/latest/management_api/

Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
---
M src/kudu/util/mini_prometheus-test.cc
1 file changed, 13 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/58/24658/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24658 )

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................


Patch Set 2:

> Thanks for looking into this Alexey! (sorry was quite short on
 > bandwidth lately regarding this issue)
 > LGTM!

Thanks a lot for review!

I didn't expect it to be fixed soon since we haven't seen it much in the pre-commit runs, etc.  So, I don't see any problem w.r.t. your attention to the issue.  However, I found that it was happening in almost all recent test runs on one of my RH9 VMs, and it turned out it was easy to fix.


-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 17:19:58 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has removed a vote on this change.

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Hello Marton Greber, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24658

to look at the new patch set (#2).

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................

KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Before this fix, one of my RHEL9.6 x86_64 build/test machines would
regularly report a failure in the MiniPrometheusTest.StartStopNoTargets
scenario when running mini_prometheus-test.  Reports looked like below:

  src/kudu/util/mini_prometheus-test.cc:206: Failure
  Failed
  Bad status: Remote error: HTTP 503

This changelist address the issue.

I also tried a more generic approach, trying to rely on the documented
'/-/ready' URL [1] by running the readiness check right before returning
from the MiniPrometheus::Start() method.  However, it seems getting
HTTP 200 for GET/HEAD requests at the '/-/ready' URL doesn't guarantee
that Scrape/Target Manager initialization is complete, so the readiness
check is useless in this context.

[1] https://prometheus.io/docs/prometheus/latest/management_api/

Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
---
M src/kudu/util/mini_prometheus-test.cc
1 file changed, 13 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/58/24658/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Marton Greber (Code Review)" <ge...@cloudera.org>.

Marton Greber has posted comments on this change. ( http://gerrit.cloudera.org:8080/24658 )

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................


Patch Set 2: Code-Review+2

Thanks for looking into this Alexey! (sorry was quite short on bandwidth lately regarding this issue)
LGTM!


-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 15:14:32 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24658 )

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................


Patch Set 2: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2846/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 00:39:19 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24658 )

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................


Patch Set 2: Verified+1

RELEASE, TSAN: unrelated failures of known flaky tests org.apache.kudu.spark.kudu.DefaultSourceTest, org.apache.kudu.backup.TestKuduBackup


-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 03:53:30 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.

Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24658 )

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................

KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Before this fix, one of my RHEL9.6 x86_64 build/test machines would
regularly report a failure in the MiniPrometheusTest.StartStopNoTargets
scenario when running mini_prometheus-test.  Reports looked like below:

  src/kudu/util/mini_prometheus-test.cc:206: Failure
  Failed
  Bad status: Remote error: HTTP 503

This changelist address the issue.

I also tried a more generic approach, trying to rely on the documented
'/-/ready' URL [1] by running the readiness check right before returning
from the MiniPrometheus::Start() method.  However, it seems getting
HTTP 200 for GET/HEAD requests at the '/-/ready' URL doesn't guarantee
that Scrape/Target Manager initialization is complete, so the readiness
check is useless in this context.

[1] https://prometheus.io/docs/prometheus/latest/management_api/

Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Reviewed-on: http://gerrit.cloudera.org:8080/24658
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Marton Greber <gr...@gmail.com>
---
M src/kudu/util/mini_prometheus-test.cc
1 file changed, 13 insertions(+), 1 deletion(-)

Approvals:
  Alexey Serbin: Verified
  Marton Greber: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>

[kudu-CR] KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets

Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.

Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24658 )

Change subject: KUDU-3772 fix flakiness in MiniPrometheusTest.StartStopNoTargets
......................................................................


Patch Set 1: Verified-1

Build Failed 

http://jenkins.kudu.apache.org/job/pre_commit/2845/ : FAILURE


-- 
To view, visit http://gerrit.cloudera.org:8080/24658
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I05e9ca3e3c8782e9b073dd2329846d5eac661dcf
Gerrit-Change-Number: 24658
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <gr...@gmail.com>
Gerrit-Comment-Date: Tue, 04 Aug 2026 23:23:21 +0000
Gerrit-HasComments: No