[jira] [Created] (KAFKA-19743) [DISCUSS] Exploring Compile-time AST Generation as an Alternative Approach for Kafka Protocol Classes

["Yingquan He (Jira)" <ji...@apache.org>]

Yingquan He created KAFKA-19743:
-----------------------------------

             Summary: [DISCUSS] Exploring Compile-time AST Generation as an Alternative Approach for Kafka Protocol Classes
                 Key: KAFKA-19743
                 URL: https://issues.apache.org/jira/browse/KAFKA-19743
             Project: Kafka
          Issue Type: Wish
            Reporter: Yingquan He


h2. 1. Background & Motivation

Currently, Kafka uses a JSON template-based pre-generation approach to create protocol classes under {{{}org.apache.kafka.common.message{}}}. This solution is mature, stable, and provides excellent multi-language support.

However, I've been considering whether we could provide an alternative *compile-time code generation* approach for the Java ecosystem, similar to the AST modification technology used by Lombok. This approach might offer better development experience in certain scenarios.
h2. 2. Proposal Overview

{*}Core Idea{*}: Use annotation processors to directly modify the AST during compilation, dynamically generating complete implementations of protocol classes instead of pre-generating Java source files.

{*}Example Implementation Concept{*}:
{code:java}
@KafkaMessage(apiKey = 1, version = "0.1.0")
public class FetchRequestSpec {
    @ProtocolField(type = "string", order = 1)
    private String groupId;
    
    @ProtocolField(type = "int16", order = 2) 
    private short acks;
    
    // More fields...
}{code}
 
h2. 3. Comparative Analysis
||Aspect||Current Template Approach||Compile-time AST Approach||Hybrid Possibility||
|*Development Experience*|Requires viewing generated source files|Cleaner source code, Lombok-like|AST during development, pre-gen for release|
|*Build Process*|Explicit pre-generation step|Integrated into standard compilation|Configurable generation strategy|
|*Debugging Support*|✅ Full source-level debugging|⚠️ Requires IDE plugin support|Source output on demand|
|*Multi-language Support*|✅ Single definition, multiple outputs|❌ Java-only|Preserve existing multi-language capabilities|
|*Performance Impact*|One-time generation at build time|Regeneration on each compilation|Smart caching mechanism|
h2. 4. Potential Value
 * {*}Faster Development Iteration{*}: Immediate compilation after protocol definition changes, no pre-generation step required

 * {*}Source Code Cleanliness{*}: Avoid polluting codebase with大量 generated boilerplate code

 * {*}Type Safety{*}: Stricter protocol constraint checking at compile time

 * {*}Alignment with Modern Java Patterns{*}: Similar to Quarkus, Micronaut's compile-time processing philosophy

h2. 5. Feasibility Considerations

I understand the potential challenges this approach might face:
 * Compatibility with existing multi-language ecosystem

 * Ensuring debugging experience

 * Increased build system complexity

 * Community learning curve

Therefore, I suggest we could:
 # {*}Offer as Optional Approach{*}: Doesn't disrupt existing workflow, serves as alternative during development

 # {*}Incremental Implementation{*}: Start with small scope or new modules for experimentation

 # {*}Toolchain Support{*}: Provide source output mode to ensure debugging capability

h2. 6. Next Steps & Recommendations

If the community finds this direction interesting, I'm willing to:
 * Create a Proof of Concept (POC) implementation to demonstrate concrete results

 * Participate in relevant discussions and design work

 * Assist with performance benchmarking and compatibility validation

 

Thank you for taking the time to review this proposal. I look forward to further discussing this idea with the community, and whether it is ultimately adopted or not, I appreciate the opportunity to share my thoughts with the community.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)