You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by "soumilshah1995 (via GitHub)" <gi...@apache.org> on 2026/03/15 20:22:57 UTC
[D] Making Hudi Github Project More Agentic [hudi]
GitHub user soumilshah1995 created a discussion: Making Hudi Github Project More Agentic
Hi everyone — I had a small proposal that could help make our open-source GitHub repos much more **AI-agent friendly** and easier to maintain.
**Idea: introduce a few lightweight repo conventions so AI agents understand the project better and can contribute safely.**
**A. `AGENTS.md`**
A file that explains the repository structure, coding conventions, and important modules so AI agents understand how the project is organized before making changes.
**B. `ARCHITECTURE.md`**
A high-level overview of the system architecture, key components, and data flow. This helps agents reason about design decisions instead of only reading code.
**C. Documentation Skill (AI-based)**
Introduce an automated documentation workflow where an AI agent periodically scans the repo and generates or updates documentation for APIs, modules, and examples. This could help keep open-source documentation consistently up to date.
**Goal:**
Make the repo easier for both humans and AI agents to understand, improve onboarding, and enable more reliable automated contributions.
Happy to experiment with a small prototype if folks think this could be useful.
GitHub link: https://github.com/apache/hudi/discussions/18324
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by Austin Bennett <wh...@gmail.com>.
Don't forget there are some great uses of Agentic Code Review as well to
make it easier on reviewers.
On Sun, Mar 15, 2026 at 10:57 PM codope (via GitHub) <gi...@apache.org> wrote:
>
> GitHub user codope added a comment to the discussion: Making Hudi Github
> Project More Agentic
>
> Yes, i think this would be very useful for the community. How do we plan
> to keep the AGENTS/ARCHITECTURE.md files updates with major changes? Also,
> if you could share some of your own experience of how you use agents with
> Hudi repo, it would be very helpful for the community.
>
> GitHub link:
> https://github.com/apache/hudi/discussions/18324#discussioncomment-16152675
>
> ----
> This is an automatically sent email for dev@hudi.apache.org.
> To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
>
>
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
GitHub user vinothchandar added a comment to the discussion: Making Hudi Github Project More Agentic
@soumilshah1995 on the original AGENTS.md/ARCHITECTURE.md/CLAUDE.md +1 . I think @yihua has a PR for CLAUDE.md file. we should consider that as well. but happy to take contributions around this. get it landed and keep chipping away
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16468114
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c added a comment to the discussion: Making Hudi Github Project More Agentic
@soumilshah1995 thanks soumil for starting the discussion! If there is something you want to take up wondering if you want to just file a github issue and I can assign to you based on the discussion we are having. Also happy to jam over slack if you want feel free to reach out!
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16521113
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
GitHub user geserdugarov added a comment to the discussion: Making Hudi Github Project More Agentic
I suppose that for keeping AGENTS/ARCHITECTURE.md we have to except eventual consistency of them. For instance, someone on weekly basis will burn his tokens, and use the latest model with the biggest context windows to pass through all merged changes, and update those files.
From my experience, the goal of managing of AI-based data should be to keep it tight. It's really easy to allow to grow it exponentially. Removing is hard these days.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16287105
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
GitHub user vinothchandar added a comment to the discussion: Making Hudi Github Project More Agentic
This can be get way sharper if we leaned in and wrote skills
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16468100
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
GitHub user codope added a comment to the discussion: Making Hudi Github Project More Agentic
Yes, i think this would be very useful for the community. How do we plan to keep the AGENTS/ARCHITECTURE.md files updates with major changes? Also, if you could share some of your own experience of how you use agents with Hudi repo, it would be very helpful for the community.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16152675
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
GitHub user soumilshah1995 added a comment to the discussion: Making Hudi Github Project More Agentic
Hi @vinothchandar — your ideas are fantastic. Having the Hudi CLI controlled by Claude sounds like a great starting point.
Also, could you share if there’s an existing CLAUDE.md in any active PR? If not, I’d be happy to help add one. We could initialize it using /init, let Claude scan the repository, and generate a baseline CLAUDE.md. From there, we can gradually evolve it by adding rules, coding standards, and best practices.
This could help make the Hudi repo more agent-friendly, and for anyone using modern AI tools, it would also encourage better documentation and consistency across contributions.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16493742
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
GitHub user soumilshah1995 added a comment to the discussion: Making Hudi Github Project More Agentic
Thanks a lot for sharing this
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16517770
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
GitHub user geserdugarov edited a comment on the discussion: Making Hudi Github Project More Agentic
I suppose that for keeping AGENTS/ARCHITECTURE.md we have to except eventual consistency of them. For instance, someone on weekly basis will burn his tokens, and use the latest model with the biggest context windows to pass through all merged changes, and update those files.
From my experience, the goal of managing of AI-based data should be to keep it tight. It's really easy to allow to grow it exponentially. The more description you have the highest probability of conflict in it. Removing is hard these days.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16287105
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
GitHub user voonhous added a comment to the discussion: Making Hudi Github Project More Agentic
See discussion here:
https://github.com/apache/hudi/discussions/19264
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-17634575
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
GitHub user geserdugarov edited a comment on the discussion: Making Hudi Github Project More Agentic
I suppose that for keeping AGENTS/ARCHITECTURE.md we have to except eventual consistency of them. For instance, someone on weekly basis will burn his tokens, and use the latest model with the biggest context windows to pass through all merged changes, and update those files.
From my experience, the goal of managing of AI-based data should be to keep it tight. It's really easy to allow to grow it exponentially. The more description you have the highest probability of conflicts in it. Removing is hard these days.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16287105
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan added a comment to the discussion: Making Hudi Github Project More Agentic
Right way to go Vinoth! We should totally build this. I can also volunteer to contribute or coordinate the effort with others.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16510348
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
GitHub user soumilshah1995 added a comment to the discussion: Making Hudi Github Project More Agentic
That's a really nice idea
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16447614
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "brucearctor (via GitHub)" <gi...@apache.org>.
GitHub user brucearctor added a comment to the discussion: Making Hudi Github Project More Agentic
sounds like agentskills.
An example:
https://temporal.io/blog/introducing-temporal-developer-skill
ex ->
https://github.com/temporalio/skill-temporal-developer/tree/main/references/java
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16452070
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan added a comment to the discussion: Making Hudi Github Project More Agentic
Let me recap a structured approach for integrating AI agents into Hudi in a practical and incremental way.
Workstream 1: Agent-Friendly Repository
Introduce lightweight, maintainable documentation to help both humans and AI agents better understand the project:
Add a concise AGENTS.md describing key concepts, workflows, and entry points
Add an ARCHITECTURE.md capturing high-level system design and core components
Optionally introduce a PR checklist item to keep these documents updated as the code evolves
The goal is to improve discoverability without introducing heavy maintenance overhead.
Workstream 2: Agent-Friendly Runtime Interface
This aligns with the broader vision of a Claude/Codex-like experience for building and operating lakehouses. (Vinoth's proposal)
Phase 1: Read-Only Investigation Tools
Create a modern Hudi operator surface (via MCP and/or an enhanced CLI) focused on safe, read-only operations:
List tables
Describe table metadata
Fetch and inspect timeline
Summarize recent commits (inserts/updates)
Show table services state (compaction, clustering, cleaning)
Explain key configurations and their impact
This phase focuses on observability and understanding, enabling users (and agents) to reason about table state safely.
Phase 2: Guided Actions (Controlled Mutations)
After validating read-only workflows and gathering community feedback, introduce controlled write operations:
Trigger compaction and clustering
Trigger cleaner and archival
Trigger rollbacks for pending/inflight ingestion commits
Suggest configuration changes based on observed patterns
Generate commands first (instead of executing directly), allowing users to review before applying
The emphasis here is on safe, guided execution, not full automation.
Workstream 3: Guided Labs and Onboarding
Provide structured, hands-on experiences to help users get started quickly:
// This is what Soumil proposed above or have already started efforts on
Create guided workflows for:
Table creation and ingestion
Incremental processing
Table services (compaction, clustering)
Optionally integrate agent-assisted walkthroughs
The goal is to reduce the learning curve and accelerate adoption.
Workstream 4: Troubleshooting and Diagnostics
Enable agents to assist with common operational issues:
Diagnose ingestion failures and stuck pipelines
Identify performance bottlenecks (e.g., small files, skew, slow queries)
Analyze timeline issues (pending/inflight commits)
Provide actionable recommendations for fixes
This workstream focuses on operational productivity and debugging efficiency.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16578617
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan added a comment to the discussion: Making Hudi Github Project More Agentic
hudi cli mcp support https://github.com/apache/hudi/pull/18554
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16678187
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
GitHub user soumilshah1995 added a comment to the discussion: Making Hudi Github Project More Agentic
Hi there, I’m planning to build a “delete streamer” lab setup that lets anyone get started in just a few minutes locally.
Of course, I’m using Claude to help with this. The idea is to make the learning process fun and easy, instead of people getting stuck worrying about knobs and configuration.
I’ve already completed around 12 labs on Delta Streamer, and now I want to take it further by building an agent-driven learning experience.
https://github.com/soumilshah1995/apache-hudi-delta-streamer-labs
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16549196
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
GitHub user cshuo added a comment to the discussion: Making Hudi Github Project More Agentic
That's a great point. For keeping `AGENTS.md` / `ARCHITECTURE.md` up to date, maybe we can use a lightweight process:
1. Add a PR checklist item: “Does this change require updates to AGENTS/ARCHITECTURE docs?”
2. Optionally add a non-blocking CI reminder when large structural changes are detected (new modules/package moves/public API changes).
3. Do a quick doc consistency pass before each release.
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16303714
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
GitHub user vinothchandar added a comment to the discussion: Making Hudi Github Project More Agentic
I think indeed this is a very important step that the Hudi community needs to take. I feel there is a fundamental gap here that an AI-native experience can bridge for the project. For many years now, I think we've been having this divide where users want both advanced, powerful features and don't want to deal with the complexity of tuning and learning configs and things like that. Without AI, this unfortunately was not possible. For any other database, you get a system that can do many things powerfully; then you also need to spend the time learning the system.
As the data lake became a lake house, you need data engineers to also level up and become kind of data lake administrators. Unfortunately, this never happened. Engineers are sold completely closed solutions even today for the sake of ease of use.
But I think AI presents us an opportunity to bridge this gap once and for all for open source. I would like to reimagine the interface for the project and for any user to build a lake house to be a Claude Code or codex-like terminal experience.
I think with lots of will and a little bit of skill, I think we can go get there much faster than we think. The vision is you create, build, optimize, deploy -- everything from a terminal agentic CLI.
For starters
- fully porting the hudi-cli to this model, with ability to connct to local fs, cloud storage locations, catalogs will help the entire community do more without breaking a sweat.
- Build some version aware (foundational models don't get this) RAG over hudi-docs, can help optimize config tuning, along with an `optimize` skill.
I built a toy MCP server with some 27 tools, over the weekend, with auto discovery of tables (local fs for now, we can bring catalogs and cloud storage in there pretty easily). I think we can even make `@hudi/table/<table_name>` work as MCP resources.
```
[13:25:58] hudi-agent % claude
▐▛███▜▌ Claude Code v2.1.92
...
...
❯ what hudi tables do I have?
⏺ hudi - hudi_list_tables (MCP)
⎿ {
"result": "Discovered 11 table(s):\n hudi_demo_table COPY_ON_WRITE /private/tmp/hudi-
agent-demo/demo_table\n customers COPY_ON_WRITE /private/tmp/hudi-datalake/custom
… +9 lines (ctrl+o to expand)
⏺ You have 11 Hudi tables across two storage locations:
┌─────────────────┬──────┬──────────────────────────────────────────┐
│ Table │ Type │ Path │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ hudi_demo_table │ COW │ /private/tmp/hudi-agent-demo/demo_table │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ customers │ COW │ /private/tmp/hudi-datalake/customers │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ shipments │ COW │ /private/tmp/hudi-datalake/shipments │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ payments │ COW │ /private/tmp/hudi-datalake/payments │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ products │ COW │ /private/tmp/hudi-datalake/products │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ user_sessions │ MOR │ /private/tmp/hudi-datalake/user_sessions │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ audit_log │ COW │ /private/tmp/hudi-datalake/audit_log │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ clickstream │ MOR │ /private/tmp/hudi-datalake/clickstream │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ inventory │ MOR │ /private/tmp/hudi-datalake/inventory │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ orders │ COW │ /private/tmp/hudi-datalake/orders │
├─────────────────┼──────┼──────────────────────────────────────────┤
│ reviews │ MOR │ /private/tmp/hudi-datalake/reviews │
└─────────────────┴──────┴──────────────────────────────────────────┘
6 are Copy-on-Write and 4 are Merge-on-Read tables. Most live under /private/tmp/hudi-datalake/. Want to inspect any of
them?
❯ Analyze layout of table clickstream
⏺ hudi - hudi_describe_table (MCP)(table_name: "clickstream")
⎿ {
"result": "[\n {\n \"Property\": \"basePath\",\n \"Value\": \"/private/tmp/hudi-datalake/clickstream\"\
n },\n {\n \"Property\": \"metaPath\",\n \"Value\": \"/private/tmp/hudi-datalake/clickstream/.hoodie\"\n
… +25 lines (ctrl+o to expand)
⏺ hudi - hudi_fetch_schema (MCP)(table_name: "clickstream")
⎿ {
"result": "{\n \"type\" : \"record\",\n \"name\" : \"clickstream_record\",\n \"namespace\" : \"hoodie.click
stream\",\n \"fields\" : [ {\n \"name\" : \"_hoodie_commit_time\",\n \"type\" : [ \"null\", \"string\" ],
… +10 lines (ctrl+o to expand)
⏺ hudi - hudi_timeline_active (MCP)(table_name: "clickstream")
⎿ {
"result": "[\n {\n \"Instant\": \"20260404203259123\",\n \"Action\": \"deltacommit\",\n \"State\": \
"COMPLETED\"\n }\n]"
}
⏺ hudi - hudi_metadata_partitions (MCP)(table_name: "clickstream")
⎿ {
"result": "[\n \"dt=2024-01-01\",\n \"dt=2024-01-02\",\n \"dt=2024-01-03\"\n]"
}
⏺ hudi - hudi_metadata_stats (MCP)(table_name: "clickstream")
⎿ {
"result": "[]"
}
⏺ hudi - hudi_commits_show (MCP)(table_name: "clickstream")
⎿ {
"result": "[\n {\n \"CommitTime\": \"20260404203259123\",\n \"Action\": \"deltacommit\",\n \"TotalBy
tesWritten\": 1307319,\n \"TotalFilesInsert\": 3,\n \"TotalFilesUpdated\": 0,\n \"Partitions\": 3,\n
… +2 lines (ctrl+o to expand)
⏺ hudi - hudi_metadata_files (MCP)(table_name: "clickstream")
⎿ {
"result": "[]"
}
⏺ hudi - hudi_compactions_show (MCP)(table_name: "clickstream")
⎿ {
"result": "[]"
}
⏺ Here's the full layout analysis for clickstream:
---
Table Identity
┌─────────────────┬────────────────────────────────────────┐
│ Property │ Value │
├─────────────────┼────────────────────────────────────────┤
│ Type │ Merge-on-Read (MOR) │
├─────────────────┼────────────────────────────────────────┤
│ Base path │ /private/tmp/hudi-datalake/clickstream │
├─────────────────┼────────────────────────────────────────┤
│ Record key │ id │
├─────────────────┼────────────────────────────────────────┤
│ Partition field │ dt (hive-style) │
├─────────────────┼────────────────────────────────────────┤
│ Ordering field │ ts │
├─────────────────┼────────────────────────────────────────┤
│ Merge mode │ EVENT_TIME_ORDERING │
├─────────────────┼────────────────────────────────────────┤
│ Key generator │ SIMPLE │
├─────────────────┼────────────────────────────────────────┤
│ Table version │ 9 │
├─────────────────┼────────────────────────────────────────┤
│ Timeline layout │ v2 │
└─────────────────┴────────────────────────────────────────┘
Schema (6 user fields + 5 meta fields)
┌─────────┬─────────────────────────────────────┐
│ Field │ Type │
├─────────┼─────────────────────────────────────┤
│ id │ int (nullable) │
├─────────┼─────────────────────────────────────┤
│ user_id │ int (nullable) │
├─────────┼─────────────────────────────────────┤
│ page │ string (nullable) │
├─────────┼─────────────────────────────────────┤
│ action │ string (nullable) │
├─────────┼─────────────────────────────────────┤
│ ts │ long (nullable) — ordering field │
├─────────┼─────────────────────────────────────┤
│ dt │ string (nullable) — partition field │
└─────────┴─────────────────────────────────────┘
Plus the standard _hoodie_commit_time, _hoodie_commit_seqno, _hoodie_record_key, _hoodie_partition_path,
_hoodie_file_name meta fields.
Partitions (3)
- dt=2024-01-01
- dt=2024-01-02
- dt=2024-01-03
Timeline & Commits
Single completed deltacommit at 20260404203259123:
- 8 records inserted across 3 partitions
- 3 log files written (one per partition — typical for MOR inserts)
- ~1.3 MB total bytes written
- 0 updates, 0 write errors
Metadata Table
Enabled indexes: files, column_stats, partition_stats
Compactions
None pending. Since there's only one deltacommit with log files and no base (parquet) files yet, the table is entirely
in delta log form. A compaction would convert the logs into columnar base files — something to consider if you plan to
run read-optimized queries.
---
Observations
1. Very young table — single commit, 8 records. Essentially a fresh table with seed data.
2. All data in log files — no base files exist yet. Snapshot queries will read logs directly; read-optimized queries
would return nothing until a compaction runs.
3. Even distribution — ~2-3 records per partition across the 3 date partitions.
4. Good metadata setup — column_stats and partition_stats are enabled, which will help with data skipping as the table
grows.
Would you like me to dig deeper into any specific aspect, or look at the file-level details within a partition?
✻ Cooked for 33s
```
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16468082
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
GitHub user soumilshah1995 edited a comment on the discussion: Making Hudi Github Project More Agentic
Hi there, I’m planning to build a “delta streamer” lab setup that lets anyone get started in just a few minutes locally.
Of course, I’m using Claude to help with this. The idea is to make the learning process fun and easy, instead of people getting stuck worrying about knobs and configuration.
I’ve already completed around 12 labs on Delta Streamer, and now I want to take it further by building an agent-driven learning experience.
https://github.com/soumilshah1995/apache-hudi-delta-streamer-labs
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16549196
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
GitHub user rahil-c added a comment to the discussion: Making Hudi Github Project More Agentic
@soumilshah1995 One idea that we were talking about in the hudi community sync that @vinothchandar brought up was the idea of revamping the current hudi-cli to embrace this new era of agents. Ideally giving `claude`, `codex` or some LLM context on the hudi project in terms of how to run some commands or even adding some new skill for operating a hudi table (such as running table services or tuning certain configs). Happy to sync at some point and talk thru some ideas with you as well!
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16417676
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] Making Hudi Github Project More Agentic [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: Making Hudi Github Project More Agentic
@soumilshah1995 Thanks for the initiatives! Here's the WIP PR on adding Claude Skills for Hudi: #18338
Do you want to create a PR on adding `CLAUDE.md`, then we can iterate over it?
GitHub link: https://github.com/apache/hudi/discussions/18324#discussioncomment-16512094
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org