You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "kennknowles (via GitHub)" <gi...@apache.org> on 2026/09/02 18:15:52 UTC
[PR] Implement LLM-driven review assigner for PR bot [beam]
kennknowles opened a new pull request, #39980:
URL: https://github.com/apache/beam/pull/39980
Beam's existing PR reviewer assigner works like so:
- matches directory paths against coarse labels (e.g. java, python, kafka)
- and assigns one reviewer per matching label via round-robin
The downsides:
- Reviewer bloat on multi-directory PRs
- Doesn't stay up to date
- Is pretty coarse
This PR is meant to introduce a little more intelligence:
- Read the history and do some heuristics on who has edited in a meaningful way recently
- Provide that heuristic along with the history to an LLM to choose a good reviewer
- Give the reviewer a one-liner about why they were chosen
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "github-actions[bot] (via GitHub) via github" <gi...@beam.apache.org>.
github-actions[bot] commented on PR #39980:
URL: https://github.com/apache/beam/pull/39980#issuecomment-6796012509
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3926451446
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
Tried it... and yes it is better, but burns tokens for a while and lots of tool calls, plus I didn't attempt to restrict the context away from the existing PR comments and reviewers.
The funniest result would be if it said "me! I'm the ultimate code reviewer assistant" but alas...
# Suggest a good reviewer for https://github.com/apache/beam/pull/39156
The best reviewer for apache/beam#39156 https://github.com/apache/beam/pull/39156 is @ahmedabu98 (Ahmed Abualsaud).
──────
### Primary Recommendation: @ahmedabu98 (Ahmed Abualsaud)
• Original Author: Authored WriteToPartitions.java https://github.com/apache/beam/blob/master/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WriteToPartitions.java in PR #38061
https://github.com/apache/beam/pull/38061 when introducing hash distribution mode for Iceberg writes.
• Component Lead: Primary maintainer and most active committer across the sdks/java/io/iceberg https://github.com/apache/beam/tree/master/sdks/java/io/iceberg module.
• Context / Current Blocker: Ahmed already reviewed this PR on June 29 and left an inline design question on WriteToPartitions.java https://github.com/apache/beam/pull/39156#discussion_r3494205093:
│ "Were you seeing OOMs even with autosharding enabled? AFAIK autosharding should distribute to different keys, so the state for any one key shouldn't get too large. If it makes sense, we can keep
│ the GroupIntoBatches.withShardedKey() path for autosharding batch writes. Otherwise maybe we can add a num_shards config to do fixed sharding per partition, and stick with GBK"
The PR is currently stalled awaiting the author's response to this question, which has caused the Beam PR bot to repeatedly cycle reviewers (@ahmedabu98 ↔ @chamikaramj ↔ @Abacn). Once addressed,
@ahmedabu98 is best positioned to approve and merge it.
──────
### Alternative Reviewers
1. @chamikaramj (Chamikara Jayalath)
• Beam PMC member and senior I/O specialist.
• Reviewed the original PR #38061 https://github.com/apache/beam/pull/38061 implementing WriteToPartitions.
• Deep expertise in Beam core transforms, batch vs. streaming execution semantics, and runner shuffle behavior (GroupByKey vs. GroupIntoBatches).
2. @clairemcginty (Claire McGinty)
• Active contributor to IcebergIO, having recently modified WriteToPartitions.java https://github.
com/apache/beam/blob/master/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WriteToPartitions.java in PR #39250 https://github.com/apache/beam/pull/39250.
• Colleague of the PR author at Spotify, with direct context on large-scale production Iceberg write pipelines.
# Suggest a good reviewer for https://github.com/apache/beam/pull/37535
Considering Stream Completion Logic
Based on the changes in PR #37535 https://github.com/apache/beam/pull/37535 ("Fix temp table cleanup for continuous read in BigQueryIO" by @stankiewicz), here are the best reviewers to suggest:
──────
### Primary Recommendations
#### 1. Yi Hu (@Abacn https://github.com/Abacn) — Recommended primary reviewer
• Why:
• Already active on this PR: Yi previously reviewed this PR in April 2026 and requested a load test regarding the overhead of the stateful CleanupTempTableDoFn, as well as handling the zero-
stream edge case.
• Addressed feedback: The author (Radek) has since conducted the benchmark, posted the load test doc https://docs.google.com/document/d/1yUxqlv2118xc-6AN-
8OxFgBi9iZRg0HcDToVElWoetY/edit?usp=sharing, and added tests for zero streams.
• Code ownership: Designated reviewer for the Java label and frequent contributor to BigQueryStorageStreamSource and BigQuery IO.
• Next step: A ping to Yi to review the benchmark results and verify the zero-stream edge case test.
#### 2. Chamikara Jayalath (@chamikaramj https://github.com/chamikaramj) — Recommended for runner/fault-tolerance validation
• Why:
• Pending review thread: Chamikara reviewed the PR in July 2026 and raised a critical question regarding runner failure and retry semantics:
│ "Should there be a shuffle between 'ReadDynamicStreamSource' and 'CleanupTempTableDoFn'? What if there's a runner failure of the downstream output processing and a retry after cleanup
│ operation already performed? ... Just want to make sure that we considered and ruled out the case of premature deletion of temp tables before they are fully processed and corresponding
│ workitems are completed."
• Radek replied on July 23 noting that CleanupTempTableDoFn is a stateful DoFn (introducing a key-by shuffle on the cleanup messages), but Chamikara has not yet weighed in on whether that fully
guarantees downstream consumers won't attempt a re-read if an uncheckpointed bundle fails.
• Chamikara's sign-off is essential to ensure this doesn't risk data loss under runner worker failure scenarios.
──────
### Secondary / Alternative Reviewers
#### 3. Danny McCormick (@damccorm https://github.com/damccorm)
• Why:
• Danny reviewed and approved Radek's original PRs that implemented this entire feature:
• #36668 https://github.com/apache/beam/pull/36668: Streaming read for BigQuery
• #36912 https://github.com/apache/beam/pull/36912: Fix retry scenario for query to table materialization
• He already possesses full context on the design and lifecycle of BigQuery dynamic streaming reads.
#### 4. Reuven Lax (@reuvenlax https://github.com/reuvenlax)
• Why:
• Primary architect and maintainer of BigQueryIO and Dataflow streaming/stateful semantics (including the original batch temp table cleanup transform PassThroughThenCleanup).
• If there is any remaining question about the stateful DoFn approach vs. stream completion guarantees across runners, Reuven's review would be authoritative.
──────
### Suggested Action
Since @Abacn and @chamikaramj are already assigned and have pending review threads on the PR, the best step is to re-ping both of them on the PR:
R: @Abacn @chamikaramj
Specifically highlighting that the load test results and the response to the runner-retry question are ready for their re-evaluation.
# Suggest a good reviewer for https://github.com/apache/beam/pull/39363
Examining artifact staging issues
The recommended reviewer for apache/beam#39363 https://github.com/apache/beam/pull/39363 is:
### Primary Recommendation: @Abacn (Yi Hu)
Why:
• Active Engagement & Verification: @Abacn is already involved with this PR. They verified the changes on Windows (running Spark portable runner tests) and commented on the PR.
• Direct Context on Artifact Staging & Windows: They reviewed and merged the closely related PR #39367 https://github.com/apache/beam/pull/39367 (handling InvalidPathException during artifact
staging on Windows) and subsequently opened #39916 https://github.com/apache/beam/pull/39916 addressing the broader Windows artifact staging filename and CI fixes.
• Committer Status: As an Apache Beam committer and PMC member, @Abacn has the necessary permissions to guide the PR to completion and merge it once open comments are resolved.
──────
### Domain Expert Alternative: @robertwb (Robert Bradshaw)
• Why: Robert is the original author of ArtifactStagingService.java https://github.com/apache/beam/blob/master/runners/java-fn-
execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactStagingService.java and leads the Fn Execution / Portability artifact management architecture. He is the best escalation
point if there are questions about cross-language artifact retrieval contracts or naming semantics across SDKs.
──────
### Context on Existing Reviews
• @Eliaaazzz (Elia Liu) already performed an in-depth review on this PR, pointing out that the Splitter pattern ([^A-Za-z-_.]]) contains a stray bracket causing base to carry unescaped characters,
and suggesting handling of control characters. While not currently a committer with merge bits, their feedback is the primary blocker @Abacn cited as needing resolution before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #39980:
URL: https://github.com/apache/beam/pull/39980#issuecomment-5514273693
@damccorm I need to give this a more thorough review, but the biggest picture is that I don't really know what it looks like for Beam's infra to make the gemini calls that are in this using a GEMINI_API_KEY or if there is an entirely different appropriate architecture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3918073539
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
And TBH I think the selected reviewers are just almost always not the right person. I'm biased because I get on there for "java" and "website" and "fallback" and realistically there's always a better choice. I don't think load balancing to people who are that vaguely related is necessarily good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3917761273
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
We could ditch gemini and just go with the heuristic score, for example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3925795719
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
Really like analysis on those 3 PRs I would give it a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3926369522
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
I reached for LLM because it is something where a quick glance at code history typically makes it very obvious who should review. It may be that all the extra parts of the system are actually worse than just saying "choose a good reviewer" and letting it do its thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3926466947
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
I'll try it on incoming stuff that doesn't have so much history, to see if it is worth adding as a requestable service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "github-actions[bot] (via GitHub) via github" <gi...@beam.apache.org>.
github-actions[bot] commented on PR #39980:
URL: https://github.com/apache/beam/pull/39980#issuecomment-6630738271
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "github-actions[bot] (via GitHub) via github" <gi...@beam.apache.org>.
github-actions[bot] closed pull request #39980: Implement LLM-driven review assigner for PR bot
URL: https://github.com/apache/beam/pull/39980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3917394363
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
Looking at the problems, do we need a full system overhaul or can we just make smaller less-invasive changes? The gemini infra is going to be painful since it will need to be a secret we regularly rotate.
> Reviewer bloat on multi-directory PRs
Could this be solved by just choosing one of the reviewers from the set of available ones (probably with a preference for code-related expertise)?
> Doesn't stay up to date
Is this really an issue? I have not seen this one cause meaningful problems since reviewers tend to opt out fairly quickly once they are out of the loop.
I think the ability to opt in/out is a pretty useful feature.
> Is pretty coarse
This is less avoidable (though we certainly could have tighter categories).
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
I haven't done a full review, but I don't think that this has the information needed to do this optimally. Specific things which aren't covered, but seem important:
1. How many times has this reviewer been assigned recently (right now we just round robin)
2. Committership
3. Does this person actually want to be a reviewer for the area/are they willing to respond in a timely manner (I can think of contributors who commit a good amount of code, but aren't necessarily ideal reviewers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3925040197
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
> The gemini infra is going to be painful since it will need to be a secret we regularly rotate.
> We could ditch gemini and just go with the heuristic score, for example.
Ok, actually I thought a bit more about this and I don't think this will be an issue - we just shouldn't use a GEMINI_API_KEY, we can just use Vertex + gcloud auth. We do this in our model handler if no API key is provided, and our ITs rely on this - https://github.com/apache/beam/blob/562273c38b0c67071d809b338fc0c45658f39e51/sdks/python/apache_beam/ml/inference/gemini_inference.py#L193
The idea has grown on me, I think my primary concern is more around the context and how we're prompting the model.
> I'm biased because I get on there for "java" and "website" and "fallback" and realistically there's always a better choice. I don't think load balancing to people who are that vaguely related is necessarily good.
Could you give examples of 1 or 2 prs as case studies of where we could do better and what would have led to a better outcome? I think the most important part of this is that we need to pick the right set of things to prioritize when choosing a reviewer. The criteria I can think of are:
1. Competence: Don't assign a reviewer who will be incapable of providing a good review.
2. Fairness: Don't assign a reviewer to too many PRs.
3. Opt-out: Allow reviewers to opt out of being assigned too many PRs (or limit the volume).
4. Affinity: Try to assign reviewers who are most active in an area.
5. Inclusion: Try to include/train reviewers who may not be the best option, but are contributing more.
I tried to order those criteria from most to least important. I think right now, the prompt fully addresses (4) and partially addresses (1)/(5). It doesn't try to address (2) or (3).
I think the current system generally accomplishes (1) and (3). It tries to accomplish (2), (4), and (5), but probably doesn't do a great job with any of them.
Fairness is the one that I think is hardest. For reference, if you look at the last calendar year and filter out both reviews done on the reviewer's own PR and dependabot reviews (usually 1-liners):
- [Yi](https://github.com/apache/beam/pulls?q=is%3Apr+created%3A%3E%3D2025-09-04+reviewed-by%3Aabacn+-author%3Aabacn+-label%3Adependencies) has reviewed the most PRs with 501
- [I](https://github.com/apache/beam/pulls?q=is%3Apr+created%3A%3E%3D2025-09-04+reviewed-by%3Adamccorm+-author%3Adamccorm+-label%3Adependencies) am second with 397
- For comparison, [you](https://github.com/apache/beam/pulls?q=is%3Apr+created%3A%3E%3D2025-09-04+reviewed-by%3Akennknowles+-author%3Akennknowles+-label%3Adependencies+) are at 105.
I did a check of other common reviewers and couldn't find anyone with more than 129 reviews
For both Yi and me, this is driven by a lot of manual assignment, so I'm not sure if the system can totally fix it. But I worry that an affinity based assigner will overassign Yi because he is very active across the codebase (and is likely the best reviewer by heuristics)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3925375759
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
> Could you give examples of 1 or 2 prs as case studies of where we could do better and what would have led to a better outcome?
I did look at a couple in the vibe coding session, but I think it deserves more than 1 or 2. I was thinking about it and making this a standalone command-line tool to begin with will make it easy to experiment with.
I like your criteria. Of those, I'm privileging affinity above all else, which I think proxies for and causes other good criteria - particularly competence and lower latency. Fairness is tricky because different people have different amounts of their attention currently focused on Beam or on a particular area.
As a good example, I know I don't actually do a zillion reviews any more (FWIW if you search for "commenter" to get less formal reviews the gap widens even further) and I think that's actually right. I do have a lot that bounce off my inbox, but I'm only a suitable reviewer for a small fraction of them. If you switch the search criteria to having a "R: " assignment or "mentions:" it shows another trend: you and Yi do far more reviews than you are assigned by the bot, whereas I do about half as many reviews as I'm assigned by the bot 😅 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3925461443
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
OK, I'm going ahead with the following:
1. Do not just overhaul things. Use a feature flag.
2. Make a commandline version.
3. use gcloud auth + vertex AI
4. make it a comment command like "assign based on git history"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3926571843
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
Let me step back for a moment. I think we need to start with what we're optimizing for; everything else falls out from that.
> I like your criteria. Of those, I'm privileging affinity above all else, which I think proxies for and causes other good criteria - particularly competence and lower latency. Fairness is tricky because different people have different amounts of their attention currently focused on Beam or on a particular area.
I disagree with prioritizing affinity above everything. I think it does proxy for competence. Its not obvious to me that it prioritizes latency - I can see an argument that more active code contributors provide faster reviews, but I'm not sure I've seen that in practice.
I also intentionally would not prioritize latency. This rewards people who do reviews quickly (or in this case, commit a bunch of code) with... more reviews. That doesn't seem right. Conversely, I am committing less code these days than I have before, but I am not particularly interested in doing fewer reviews. I think this is a pretty natural directional progression over time (shifting workload toward less code, more reviews).
I agree that fairness is tricky, but I don't think we can ignore it - I think it is highly important, and was one of the core goals of the project initially. I think we should move towards optimizing for it more, not less. There are alternative ways of achieving this (e.g. surfacing review metrics, making those contributions more visible, making it easier for reviewers to opt out), but I'd also love to incorporate it into this bot.
I think we need to align on what we're prioritizing, and then we can figure out the how. I agree that an LLM-based solution will likely help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3925419079
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
Here's unedited agent response. I asked it to select 3 PR with at least 3 "R: " bot assignments due to slow review. My take is that... it isn't that good :-).
FWIW I'm deliberately taking the blameless premise that if someone didn't respond to the "R: " assignment then they had a good reason (too busy, not the right reviewer, the PR was not ready, etc). There could, of course, be other reasons that are less blameless, but I don't think it is productive. In any case being strictly pragmatic makes sense, in the sense of "if they didn't do it, they didn't do it and we change the system not the people".
----
Here are 3 real Apache Beam pull requests that experienced repeated reviewer re-assignments under the legacy bot without receiving substantive reviews, along with the
results of running our new review assigner on each of them.
──────
### PR 1: #39156 https://github.com/apache/beam/pull/39156 — [IcebergIO] Use GroupByKey for bounded inputs to avoid OOM
#### What happened with the legacy bot:
Because the PR touched Java code, the legacy bot attached label java and assigned reviewers via generic round-robin. It was re-assigned 6 times over 2 months:
1. 2026-06-29: R: @ahmedabu98 for label java
2. 2026-07-09: R: @chamikaramj for label java (slow review reassignment)
3. 2026-07-21: R: @ahmedabu98 for label java (slow review reassignment)
4. 2026-07-31: R: @chamikaramj for label java (slow review reassignment)
5. 2026-08-12: R: @Abacn for label java (slow review reassignment)
6. 2026-08-26: R: @ahmedabu98 for label java (slow review reassignment)
#### Running the new tool:
• File touched: sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WriteToPartitions.java
• Git history discovered:
• [2026-07-09] Claire McGinty: (IcebergIO) bugfix: propagate config properties to RecordWriter (#39250)
• [2026-05-04] Ahmed Abualsaud: [IcebergIO] Support hash distribution mode when writing rows (#38061)
### 🧭 Reviewer Assignment
- R: @clairem@spotify.com (**Primary Reviewer**)
*Expertise:* Active contributor to IcebergIO WriteToPartitions; recently implemented config property propagation for RecordWriter.
*Selected a minimal reviewer set to keep review focused. Backup expert(s): @ahmedabu98*
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
• Why this is better: Instead of cycling through general Java reviewers who don't work on Iceberg, it pinpoints Claire McGinty (who actively maintains the partition
writing logic in this exact file) with Ahmed as backup.
──────
### PR 2: #37535 https://github.com/apache/beam/pull/37535 — Fix temp table cleanup for continuous read in BigQueryIO
#### What happened with the legacy bot:
This PR bounced through 12 consecutive bot re-assignments across 6 months without an in-depth review:
• Assigned @Abacn → @chamikaramj → @kennknowles → @chamikaramj → @ahmedabu98 → @Abacn → @ahmedabu98 → @chamikaramj → @ahmedabu98 → @chamikaramj → @kennknowles →
@ahmedabu98.
• The bot blindly cycled through committers configured under the general java label.
#### Running the new tool:
• Files touched:
• sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
• sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
• sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/CleanupTempTableDoFnTest.java
• Git history discovered:
• Automatically filtered out the PR author (@stankiewicz / Radosław Stankiewicz).
• Evaluated contributors touching BigQuery storage read streams: claudevdm (lakehouse storage reads), Yi Hu (stream splitting & error handling), and reuvenlax.
### 🧭 Reviewer Assignment
- R: @claudevdm (**Primary Reviewer**)
*Expertise:* Contributor to BigQueryIO storage read streams and table schema handling.
*Selected a minimal reviewer set to keep review focused. Backup expert(s): @yathu@google.com, @relax@google.com*
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
• Why this is better: The PR specifically modifies continuous reading and temp table management in the BigQuery Storage API. Rather than looping through the 20+
generic Java reviewers, it identifies reviewers with direct commit history in BigQueryStorageStreamSource and BigQuery storage reads.
──────
### PR 3: #39363 https://github.com/apache/beam/pull/39363 — Fix artifact staging filenames on Windows
#### What happened with the legacy bot:
Because the files were under runners/java-fn-execution/, no path pattern in .github/autolabeler.yml matched. The bot fell back to assigning a fallback committer:
• It assigned @chamikaramj 7 times in a row:
R: @chamikaramj added as fallback since no labels match configuration
• It then assigned @damccorm on the 8th attempt as fallback.
#### Running the new tool:
• Files touched:
• runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactStagingService.java
• runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ArtifactStagingServiceTest.java
• Git history discovered:
• [2026-09-01] Elia Liu: Fail fast in artifact staging when storing an artifact fails (#39367) (modified both the service and the test).
• Mechanical changes (ErrorProne fixes by Radosław, Spotless bumps by Derrick) were deprioritized.
### 🧭 Reviewer Assignment
- R: @Elialiu760317@outlook.com (**Primary Reviewer**)
*Expertise:* Author of artifact staging error handling and test suite in ArtifactStagingServiceTest.
*Selected a minimal reviewer set to keep review focused. Backup expert(s): @yathu@google.com*
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
• Why this is better: The legacy bot completely failed here because no label was configured for runners/java-fn-execution, repeatedly burdening a fallback committer.
The new assigner looked directly at ArtifactStagingService.java's history and immediately selected Elia Liu, who had just touched the exact staging logic and test
file.
──────
### Summary of Improvements Demonstrated
PR | Issue with Old Bot | Old Bot Behavior | New Tool Assignment
---------------------------|--------------------------|--------------------------------------------------|------------------------------------------------------------
#39156 (IcebergIO) | Label too broad (java) | Bounced 6 times between 3 general Java reviewers | @clairem (authored Iceberg partition writer)
#37535 (BigQueryIO) | Label too broad (java) | Bounced 12 times in a 6-month loop | @claudevdm / @yathu (BigQuery storage stream contributors)
#39363 (Artifact Staging) | Missing directory label | Assigned fallback reviewer 8 times in a row | @Elialiu (authored staging error handling in that file)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement LLM-driven review assigner for PR bot [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #39980:
URL: https://github.com/apache/beam/pull/39980#discussion_r3925842888
##########
scripts/ci/pr-bot/shared/geminiReviewerAdvisor.ts:
##########
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import {
+ PrHistoryContext,
+ CandidateContributor,
+ TouchedFileContext,
+} from "./gitHistory";
+
+/**
+ * Interface representing an individual recommended reviewer.
+ */
+export interface ReviewerRecommendation {
+ readonly username: string;
+ readonly role: "primary" | "secondary";
+ readonly isCommitter: boolean;
+ readonly expertise: string;
+ readonly coveredFiles: readonly string[];
+}
+
+/**
+ * Interface representing an alternate reviewer suggestion.
+ */
+export interface AlternateReviewer {
+ readonly username: string;
+ readonly expertise: string;
+}
+
+/**
+ * Result structure produced by the reviewer advisor.
+ */
+export interface ReviewerAdviceResult {
+ readonly selectedReviewers: readonly ReviewerRecommendation[];
+ readonly alternateReviewers: readonly AlternateReviewer[];
+ readonly reasoning: string;
+ readonly source: "gemini" | "heuristic-fallback";
+}
+
+/**
+ * Interface for LLM clients that can generate structured JSON.
+ */
+export interface IGeminiClient {
+ generateJson<T>(prompt: string): Promise<T>;
+}
+
+/**
+ * Standard HTTP Gemini client using global fetch.
+ */
+export class GeminiClient implements IGeminiClient {
+ private readonly apiKey: string;
+ private readonly model: string;
+
+ constructor(apiKey: string, model: string = "gemini-2.5-flash") {
+ this.apiKey = apiKey;
+ this.model = model;
+ }
+
+ async generateJson<T>(prompt: string): Promise<T> {
+ if (!this.apiKey) {
+ throw new Error("GEMINI_API_KEY is not configured.");
+ }
+
+ const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(
+ this.model
+ )}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
+
+ const response = await fetch(url, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ },
+ body: JSON.stringify({
+ contents: [
+ {
+ role: "user",
+ parts: [{ text: prompt }],
+ },
+ ],
+ generationConfig: {
+ temperature: 0.1,
+ responseMimeType: "application/json",
+ },
+ }),
+ });
+
+ if (!response.ok) {
+ const errorText = await response.text();
+ throw new Error(
+ `Gemini API request failed with status ${response.status}: ${errorText}`
+ );
+ }
+
+ const data: any = await response.json();
+ const candidateText = data?.candidates?.[0]?.content?.parts?.[0]?.text;
+
+ if (!candidateText) {
+ throw new Error("Empty or invalid candidate response from Gemini API.");
+ }
+
+ return JSON.parse(candidateText) as T;
+ }
+}
+
+/**
+ * Configuration options for the Gemini Reviewer Advisor.
+ */
+export interface ReviewerAdvisorOptions {
+ readonly geminiClient?: IGeminiClient;
+ readonly committerCheck?: (username: string) => Promise<boolean>;
+ readonly exclusionList?: readonly string[];
+ readonly maxReviewers?: number;
+}
+
+/**
+ * Advisor that analyzes PR git history and selects optimal reviewers using Gemini or heuristic fallback.
+ */
+export class GeminiReviewerAdvisor {
+ private readonly client?: IGeminiClient;
+ private readonly committerCheck: (username: string) => Promise<boolean>;
+ private readonly exclusionList: readonly string[];
+ private readonly maxReviewers: number;
+
+ constructor(options: ReviewerAdvisorOptions = {}) {
+ this.client = options.geminiClient;
+ this.committerCheck = options.committerCheck ?? (async () => false);
+ this.exclusionList = options.exclusionList ?? [];
+ this.maxReviewers = options.maxReviewers ?? 2;
+ }
+
+ /**
+ * Constructs the prompt instructing Gemini on how to select reviewers.
+ *
+ * @param context Extracted git and PR history.
+ * @param committers Map of username to committer status.
+ * @returns Detailed prompt string.
+ */
+ public buildPrompt(
+ context: PrHistoryContext,
+ committers: Readonly<Record<string, boolean>>
+ ): string {
+ const fileSummaries = context.touchedFiles.map((file) => {
+ const commitSummaries = file.recentCommits
+ .slice(0, 5)
+ .map(
+ (c) =>
+ ` - [${c.date}] ${c.authorLogin || c.authorName}: ${c.subject}`
+ )
+ .join("\n");
+
+ return `- File: ${file.path} (+${file.additions}, -${
+ file.deletions
+ }, changes: ${file.changes}${
+ file.isNewFile ? " [NEW FILE]" : ""
+ })\n Recent Commits:\n${commitSummaries || " (No recent commits)"}`;
+ });
+
+ const candidateSummaries = context.candidates.map((c) => {
+ const isCommitter = committers[c.login] ?? false;
+ return `- @${c.login} (${c.name}): ${
+ c.commitCount
+ } commits, last active ${
+ c.lastCommitDate
+ }, committer=${isCommitter}. Files touched: ${c.touchedFilePaths.join(
+ ", "
+ )}`;
+ });
+
+ const exclusions =
+ this.exclusionList.map((e) => `@${e}`).join(", ") || "(none)";
+
+ return `You are the Apache Beam Code Review Assigner.
+Your goal is to choose a small, optimal set of expert reviewers for a pull request based on real git history and file churn.
Review Comment:
FWIW, out of the 3 examples you gave, I think the current bot assigned a good reviewer in the first 2. It clearly got stuck on the third one (that just looks like a bug), but I don't think the LLM solution (assigning a GSoC contributor) is better. It is also unclear who a good reviewer there is (Cham is probably as good as anyone).
> FWIW I'm deliberately taking the blameless premise that if someone didn't respond to the "R: " assignment then they had a good reason (too busy, not the right reviewer, the PR was not ready, etc). There could, of course, be other reasons that are less blameless, but I don't think it is productive. In any case being strictly pragmatic makes sense, in the sense of "if they didn't do it, they didn't do it and we change the system not the people".
I do think this is a problem worth addressing. Ideally we would have better expectations here for what is reasonable. I'll also note that one of the big problems with the current setup is distributed ownership (if 3 reviewers are assigned to a PR, no reviewers are assigned to the PR)
In my experience, it is more common to have a qualified reviewer who doesn't review in a timely manner than to have a set of reviewers with nobody capable of providing a review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org