You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "szehon-ho (via GitHub)" <gi...@apache.org> on 2026/08/06 00:43:38 UTC
[PR] [MINOR][DOC] Suggest where to place new class members in AGENTS.md [spark]
szehon-ho opened a new pull request, #57803:
URL: https://github.com/apache/spark/pull/57803
### What changes were proposed in this pull request?
Adds one paragraph to the `## Development Notes` section of `AGENTS.md` suggesting where to place a new member when editing an existing class: a private helper is better placed after the public methods that use it, or with the file's existing helpers, rather than inserted between two public methods or between two `test(...)` blocks.
It is phrased as a suggestion with a default, not a rule. Files in the repo vary in how they order members, so the note explicitly defers to the local convention of whatever file is being edited and does not ask anyone to reorganize existing code.
### Why are the changes needed?
Agents writing code top-down tend to define a helper immediately above the first method that uses it, which lands it in the middle of a class's public surface or a suite's list of test cases and interrupts a top-to-bottom read. I hit exactly this while writing a test suite and had to move the helper in review.
This fits the existing pattern of the surrounding notes, which call out specific, recurring failure modes (non-ASCII characters in comments, overrunning the 100-character line limit) rather than restating general style guidance. Unlike those two, this one has no linter behind it, so the note is the only signal available.
### Does this PR introduce _any_ user-facing change?
No. Documentation for contributors and agent tooling only.
### How was this patch tested?
No tests; documentation only. Verified the added text is pure ASCII, per the rule in the paragraph immediately below it:
```
git diff -U0 -- AGENTS.md | grep '^+' | grep -v '^+++' | grep -P "[^\x00-\x7F]"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest where to place new class members in AGENTS.md [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57803:
URL: https://github.com/apache/spark/pull/57803#discussion_r3731287876
##########
AGENTS.md:
##########
@@ -18,6 +18,8 @@ SQL golden file tests are managed by `SQLQueryTestSuite` and its variants. Read
Spark Connect protocol is defined in proto files under `sql/connect/common/src/main/protobuf/`. Read the README there before modifying proto definitions.
+When adding a member to an existing class, prefer a position that does not split up what is already there. A good default for a new private helper is after the public methods that use it, or alongside the file's existing helpers; in a test suite, after the `test(...)` blocks rather than between them. The common failure mode is placing a helper immediately above the first method that uses it, which interrupts a top-to-bottom read of the class's public surface or of the suite's list of cases. Conventions vary between files, so follow the local one rather than reorganizing.
Review Comment:
Good idea -- updated to a more general section-grouping principle: keep related kinds of code in sections and append a new member to the matching section, rather than a rule specific to private helpers. Kept a short note on the common failure mode of inserting a member next to its first use and splitting the surrounding section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest grouping code into sections when adding class members in AGENTS.md [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #57803: [MINOR][DOC] Suggest grouping code into sections when adding class members in AGENTS.md
URL: https://github.com/apache/spark/pull/57803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest where to place new class members in AGENTS.md [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57803:
URL: https://github.com/apache/spark/pull/57803#discussion_r3727636295
##########
AGENTS.md:
##########
@@ -18,6 +18,8 @@ SQL golden file tests are managed by `SQLQueryTestSuite` and its variants. Read
Spark Connect protocol is defined in proto files under `sql/connect/common/src/main/protobuf/`. Read the README there before modifying proto definitions.
+When adding a member to an existing class, prefer a position that does not split up what is already there. A good default for a new private helper is after the public methods that use it, or alongside the file's existing helpers; in a test suite, after the `test(...)` blocks rather than between them. The common failure mode is placing a helper immediately above the first method that uses it, which interrupts a top-to-bottom read of the class's public surface or of the suite's list of cases. Conventions vary between files, so follow the local one rather than reorganizing.
Review Comment:
shall we turn it into a more general code placement principle? e.g. we should group the code into sections
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest grouping code into sections when adding class members in AGENTS.md [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57803:
URL: https://github.com/apache/spark/pull/57803#issuecomment-5250507160
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/e5a596325fc190749ae1e69895ab93ef4fe43b3c
- merged into branch-4.x https://github.com/apache/spark/commit/d01fa3fe8349466df0d0ffc66d2459d330f6c6ed
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest grouping code into sections when adding class members in AGENTS.md [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57803:
URL: https://github.com/apache/spark/pull/57803#discussion_r3752417720
##########
AGENTS.md:
##########
@@ -18,6 +18,8 @@ SQL golden file tests are managed by `SQLQueryTestSuite` and its variants. Read
Spark Connect protocol is defined in proto files under `sql/connect/common/src/main/protobuf/`. Read the README there before modifying proto definitions.
+When adding members to an existing class or object, keep related kinds of code in sections rather than inserting into the middle of another section. A common layout is fields and constructors first, then the public/override API, then private helpers; in a test suite, tests together and helpers after them. Prefer appending to the matching section, or following the file's existing sectioning -- the common failure mode is inserting a new member next to its first use and splitting the surrounding section. Do not reorganize existing members unless the change requires it.
Review Comment:
Good call -- I dropped that clause. It turned out to describe a minority of suites, and the conflict argument holds up.
I scanned every `*Suite.scala` under the test trees, split each file into top-level type bodies, and bucketed each non-lifecycle helper `def` by its position relative to the test blocks. 1,174 type bodies define at least one helper:
| Helper placement | Share |
|---|---:|
| Above the first test | 49.2% |
| Interleaved among the tests | 27.7% |
| After the last test | 17.2% |
| Split across both ends | 5.9% |
So "helpers after the tests" was describing about one suite in six. And as you point out, since new tests get appended at the end, helpers parked there sit on exactly the lines new tests want.
Broken down by suite size (n = 900 small, 274 large):
| Helper placement | Small (<20 tests) | Large (20+ tests) |
|---|---:|---:|
| Above the first test | 55.6% | 28.5% |
| Interleaved among the tests | 17.7% | 60.6% |
| After the last test | 20.1% | 7.7% |
| Split across both ends | 6.7% | 3.3% |
Small suites overwhelmingly put helpers at the top; large ones mostly interleave, keeping a helper beside the group of tests that uses it. Probably two things going on: in a multi-thousand-line suite a single top block is impractical to read against, and large suites also accumulate edits from many contributors over the years, so they drift rather than following any one convention.
The style guides turned out to say less than I had assumed. Spark's contributing page points at the official Scala style guide and the Databricks guide, preferring the latter. Databricks has a single sentence on this, under "Ordering within a Class": *"If a class is long and has many methods, group them logically into different sections, and use comment headers to organize them."* No ordering beyond that. The official Scala guide's "Ordering Of Class Elements" adds only *"Fields should precede methods in a scope."* Neither mentions public-before-private, and neither says anything about test helpers. Nothing in scalastyle or scalafmt enforces member order either.
So I cut the ordering I had made up and kept only the section-grouping principle you suggested, which is the part the Databricks guide actually states:
> When adding a member to an existing class or object, follow the sectioning the file already uses and put the new member with the code it belongs with. The common failure mode is dropping it wherever it is first used without checking how the file is organized, splitting a section of unrelated code in the process. Beyond grouping related code there is no prescribed order -- the Databricks Scala guide, which Spark follows, asks only that a long class group its members into logical sections with comment headers. Do not reorganize existing members unless the change requires it.
I also reworded the failure mode. Calling it "inserting the member next to its first use" contradicted both the first sentence, since a caller is related code, and the data above, since placing a helper beside the tests it serves is the majority pattern in large suites. The real problem is placing a member without looking at how the file is organized, so that it splits an unrelated section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest where to place new class members in AGENTS.md [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #57803:
URL: https://github.com/apache/spark/pull/57803#issuecomment-5199093481
@cloud-fan @HyukjinKwon do you have any thought on it? i sometimes find annoying the ai generate a private method and put it on top
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest where to place new class members in AGENTS.md [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57803:
URL: https://github.com/apache/spark/pull/57803#issuecomment-5202505748
I don't have a strong opinion, but it makes sense to me and can be nicer to have this, so +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][DOC] Suggest grouping code into sections when adding class members in AGENTS.md [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57803:
URL: https://github.com/apache/spark/pull/57803#discussion_r3749561216
##########
AGENTS.md:
##########
@@ -18,6 +18,8 @@ SQL golden file tests are managed by `SQLQueryTestSuite` and its variants. Read
Spark Connect protocol is defined in proto files under `sql/connect/common/src/main/protobuf/`. Read the README there before modifying proto definitions.
+When adding members to an existing class or object, keep related kinds of code in sections rather than inserting into the middle of another section. A common layout is fields and constructors first, then the public/override API, then private helpers; in a test suite, tests together and helpers after them. Prefer appending to the matching section, or following the file's existing sectioning -- the common failure mode is inserting a new member next to its first use and splitting the surrounding section. Do not reorganize existing members unless the change requires it.
Review Comment:
can you ask AI to do more research? e.g. `tests together and helpers after them` looks inefficient. People usually add new test cases in the end, so putting helpers at the end is easier to cause conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org