You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Wenchen Fan <cl...@gmail.com> on 2025/11/06 16:25:15 UTC
Re: [DISCUSS] SPIP: The metrics & semantic modeling in Spark
Thanks for the proposal! I believe this is a very useful feature, as the
other alternatives do not work well: people need to either define many
similar views with different grouping columns and aggregate functions, or
manually maintain a doc page to describe the semantic of these metrics that
people need to follow when writing queries to calculate these metrics.
Shall we start the vote next week if there is no objections?
On Fri, Oct 31, 2025 at 2:30 PM Linhong Liu
<li...@databricks.com.invalid> wrote:
> Hi all,
>
> I would like to propose introducing "The metrics & semantic modeling in
> Spark".
>
> This feature enables defining business metrics once and reusing them
> across any breakdown, ensuring consistent outcomes and bridging the
> semantic gap between business logic and data schemas to help LLMs generate
> more precise results.
>
> Looking forward to your feedback!
>
> JIRA: SPARK-54119 <https://issues.apache.org/jira/browse/SPARK-54119>
> SPIP docs:
> https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc
> <https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc>
>
> Thanks,
> Linhong
>
Re: [DISCUSS] SPIP: The metrics & semantic modeling in Spark
Posted by Xiao Li <ga...@gmail.com>.
+1
This is a great direction. It provides reusable, consistent metric
definitions will really help bridge the semantic layer and make Spark more
accessible for analytical and LLM-driven use cases.
Wenchen Fan <cl...@gmail.com> 于2025年11月6日周四 10:47写道：
> Hi Anand,
>
> Thanks for the review! You are right that making metric view the single
> source of truth brings better user experience, and also allows better
> governance and LLM integration. However, I think how to handle the
> integration is out of the scope of Spark, and should be taken care by the
> vendors who provide these infrastructure.
>
> Thanks,
> Wenchen
>
> On Thu, Nov 6, 2025 at 9:57 AM Anand Chinnakannan <an...@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> I’ve reviewed and updated our Metric View proposal (Q1–Q8) to include two
>> key enhancements:
>>
>> 🔐 Governance Integration: Metric Views are now treated as governed,
>> first-class catalog objects with access control, lineage tracking, and
>> versioning — ensuring metrics remain secure, auditable, and consistently
>> defined.
>>
>> 🤖 LLM Agent Integration: Added guidance on how LLMs and AI agents can
>> discover and query metric views through catalog metadata for consistent,
>> governed responses to natural-language queries.
>>
>>
>> These updates align with our goal of making Metric Views the single
>> source of truth for analytical and AI-driven use cases.
>>
>> I’d love your input on these sections — especially around:
>>
>> 1. Any additional governance scenarios we should consider.
>>
>>
>> 2. LLM integration edge cases or optimization ideas.
>>
>>
>> 3. Suggestions for examples, syntax, or long-term roadmap points.
>>
>>
>> Please feel free to add comments, edits, or examples directly in the
>> document, or share your thoughts in reply.
>> Your contributions will help us finalize a stronger, more complete
>> proposal for review.
>>
>> Thank you for your time and collaboration — looking forward to your
>> insights!
>>
>> Best regards,
>> Anand Chinnakannan
>> Staff Data Scientist | Walmart
>> Executive MBA Candidate, Quantic School of Business & Technology
>> 📧 anandc.usa@gmail.com
>>
>> On Thu, Nov 6, 2025, 10:27 AM Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> Thanks for the proposal! I believe this is a very useful feature, as the
>>> other alternatives do not work well: people need to either define many
>>> similar views with different grouping columns and aggregate functions, or
>>> manually maintain a doc page to describe the semantic of these metrics that
>>> people need to follow when writing queries to calculate these metrics.
>>>
>>> Shall we start the vote next week if there is no objections?
>>>
>>> On Fri, Oct 31, 2025 at 2:30 PM Linhong Liu
>>> <li...@databricks.com.invalid> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I would like to propose introducing "The metrics & semantic modeling in
>>>> Spark".
>>>>
>>>> This feature enables defining business metrics once and reusing them
>>>> across any breakdown, ensuring consistent outcomes and bridging the
>>>> semantic gap between business logic and data schemas to help LLMs generate
>>>> more precise results.
>>>>
>>>> Looking forward to your feedback!
>>>>
>>>> JIRA: SPARK-54119 <https://issues.apache.org/jira/browse/SPARK-54119>
>>>> SPIP docs:
>>>> https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc
>>>> <https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc>
>>>>
>>>> Thanks,
>>>> Linhong
>>>>
>>>
Re: [DISCUSS] SPIP: The metrics & semantic modeling in Spark
Posted by Wenchen Fan <cl...@gmail.com>.
Hi Anand,
Thanks for the review! You are right that making metric view the single
source of truth brings better user experience, and also allows better
governance and LLM integration. However, I think how to handle the
integration is out of the scope of Spark, and should be taken care by the
vendors who provide these infrastructure.
Thanks,
Wenchen
On Thu, Nov 6, 2025 at 9:57 AM Anand Chinnakannan <an...@gmail.com>
wrote:
> Hi Team,
>
> I’ve reviewed and updated our Metric View proposal (Q1–Q8) to include two
> key enhancements:
>
> 🔐 Governance Integration: Metric Views are now treated as governed,
> first-class catalog objects with access control, lineage tracking, and
> versioning — ensuring metrics remain secure, auditable, and consistently
> defined.
>
> 🤖 LLM Agent Integration: Added guidance on how LLMs and AI agents can
> discover and query metric views through catalog metadata for consistent,
> governed responses to natural-language queries.
>
>
> These updates align with our goal of making Metric Views the single source
> of truth for analytical and AI-driven use cases.
>
> I’d love your input on these sections — especially around:
>
> 1. Any additional governance scenarios we should consider.
>
>
> 2. LLM integration edge cases or optimization ideas.
>
>
> 3. Suggestions for examples, syntax, or long-term roadmap points.
>
>
> Please feel free to add comments, edits, or examples directly in the
> document, or share your thoughts in reply.
> Your contributions will help us finalize a stronger, more complete
> proposal for review.
>
> Thank you for your time and collaboration — looking forward to your
> insights!
>
> Best regards,
> Anand Chinnakannan
> Staff Data Scientist | Walmart
> Executive MBA Candidate, Quantic School of Business & Technology
> 📧 anandc.usa@gmail.com
>
> On Thu, Nov 6, 2025, 10:27 AM Wenchen Fan <cl...@gmail.com> wrote:
>
>> Thanks for the proposal! I believe this is a very useful feature, as the
>> other alternatives do not work well: people need to either define many
>> similar views with different grouping columns and aggregate functions, or
>> manually maintain a doc page to describe the semantic of these metrics that
>> people need to follow when writing queries to calculate these metrics.
>>
>> Shall we start the vote next week if there is no objections?
>>
>> On Fri, Oct 31, 2025 at 2:30 PM Linhong Liu
>> <li...@databricks.com.invalid> wrote:
>>
>>> Hi all,
>>>
>>> I would like to propose introducing "The metrics & semantic modeling in
>>> Spark".
>>>
>>> This feature enables defining business metrics once and reusing them
>>> across any breakdown, ensuring consistent outcomes and bridging the
>>> semantic gap between business logic and data schemas to help LLMs generate
>>> more precise results.
>>>
>>> Looking forward to your feedback!
>>>
>>> JIRA: SPARK-54119 <https://issues.apache.org/jira/browse/SPARK-54119>
>>> SPIP docs:
>>> https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc
>>> <https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc>
>>>
>>> Thanks,
>>> Linhong
>>>
>>
Re: [DISCUSS] SPIP: The metrics & semantic modeling in Spark
Posted by Anand Chinnakannan <an...@gmail.com>.
Hi Team,
I’ve reviewed and updated our Metric View proposal (Q1–Q8) to include two
key enhancements:
🔐 Governance Integration: Metric Views are now treated as governed,
first-class catalog objects with access control, lineage tracking, and
versioning — ensuring metrics remain secure, auditable, and consistently
defined.
🤖 LLM Agent Integration: Added guidance on how LLMs and AI agents can
discover and query metric views through catalog metadata for consistent,
governed responses to natural-language queries.
These updates align with our goal of making Metric Views the single source
of truth for analytical and AI-driven use cases.
I’d love your input on these sections — especially around:
1. Any additional governance scenarios we should consider.
2. LLM integration edge cases or optimization ideas.
3. Suggestions for examples, syntax, or long-term roadmap points.
Please feel free to add comments, edits, or examples directly in the
document, or share your thoughts in reply.
Your contributions will help us finalize a stronger, more complete proposal
for review.
Thank you for your time and collaboration — looking forward to your
insights!
Best regards,
Anand Chinnakannan
Staff Data Scientist | Walmart
Executive MBA Candidate, Quantic School of Business & Technology
📧 anandc.usa@gmail.com
On Thu, Nov 6, 2025, 10:27 AM Wenchen Fan <cl...@gmail.com> wrote:
> Thanks for the proposal! I believe this is a very useful feature, as the
> other alternatives do not work well: people need to either define many
> similar views with different grouping columns and aggregate functions, or
> manually maintain a doc page to describe the semantic of these metrics that
> people need to follow when writing queries to calculate these metrics.
>
> Shall we start the vote next week if there is no objections?
>
> On Fri, Oct 31, 2025 at 2:30 PM Linhong Liu
> <li...@databricks.com.invalid> wrote:
>
>> Hi all,
>>
>> I would like to propose introducing "The metrics & semantic modeling in
>> Spark".
>>
>> This feature enables defining business metrics once and reusing them
>> across any breakdown, ensuring consistent outcomes and bridging the
>> semantic gap between business logic and data schemas to help LLMs generate
>> more precise results.
>>
>> Looking forward to your feedback!
>>
>> JIRA: SPARK-54119 <https://issues.apache.org/jira/browse/SPARK-54119>
>> SPIP docs:
>> https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc
>> <https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc>
>>
>> Thanks,
>> Linhong
>>
>