You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "nchammas (via GitHub)" <gi...@apache.org> on 2024/01/16 16:25:32 UTC
[PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]
nchammas opened a new pull request, #44755:
URL: https://github.com/apache/spark/pull/44755
### What changes were proposed in this pull request?
Enable Spark configs to be assigned to documentation groups. These groups will be used to automatically build config tables for display in our documentation.
Instead of having to maintain [large blocks of HTML tables][1] throughout our documentation, config tables can simply be included as follows:
```liquid
{% include_api_gen _generated/config_tables/sql-tuning-caching-data.html %}
```
This is a stripped down version of #44300 to facilitate review of the core idea.
[1]: https://github.com/apache/spark/blob/7db85642600b1e3b39ca11e41d4e3e0bf1c8962b/docs/sql-performance-tuning.md?plain=1#L37-L56
### Why are the changes needed?
Using this approach we can accomplish several goals at once:
- Eliminate thousands of lines of manually maintained HTML tables of Spark configs.
- Ensure that internal configs are not accidentally documented publicly. (e.g. `spark.sql.files.openCostInBytes`)
- Ensure that configs are documented publicly exactly as they are in the code. (e.g. `spark.sql.autoBroadcastJoinThreshold`)
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No testing beyond what was already done in #44300, pending review.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #44755:
URL: https://github.com/apache/spark/pull/44755#issuecomment-2098662588
Following up regarding your feedback on the API change, @holdenk. Do you recall what it was?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #44755:
URL: https://github.com/apache/spark/pull/44755#issuecomment-2078593953
> I like it, one small note around the API change
I'm missing the note. Did a line comment on the diff get swallowed up, perhaps?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #44755:
URL: https://github.com/apache/spark/pull/44755#issuecomment-1979311637
@holdenk - This is the config documentation approach we discussed on the mailing list. (The alternative, YAML-based approach is over on #44756.)
This PR just adds the fields and methods we need on `ConfigEntry`. All the work to update our documentation scripts and migrate the HTML tables into the appropriate config entries will happen in subsequent PRs. I'll group them up under an umbrella Jira ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #44755:
URL: https://github.com/apache/spark/pull/44755#issuecomment-2037640463
Oh yes, sorry it's been a busy quarter. Let me try and schedule some review time tomorrow / next week (and do please feel free to ping me again if I forget, work is just very busy as of late).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]
Posted by "nchammas (via GitHub)" <gi...@apache.org>.
nchammas commented on PR #44755:
URL: https://github.com/apache/spark/pull/44755#issuecomment-2037633304
@holdenk - Friendly ping. Are you still interested in shepherding this work? No hard feelings if not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-46395][CORE] Assign Spark configs to groups for use in documentation [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #44755:
URL: https://github.com/apache/spark/pull/44755#issuecomment-2078345591
I like it, one small note around the API change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org