[I] Databricks: support Genie consultations and resumable standalone tasks [airflow]

["zozo123 (via GitHub)" <gi...@apache.org>]

zozo123 opened a new issue, #74194:
URL: https://github.com/apache/airflow/issues/74194

   ### Description
   Track a Databricks Genie integration delivered in two bounded PRs: a hook plus managed-agent toolset adapter, then a standalone operator with deferrable waiting where supported.
   
   ### Use case/motivation
   A worker-owned agent may need a bounded consultation with a configured Genie space. A standalone DAG task may instead wait for a longer conversation/message operation. These have different resource and recovery requirements but should share authentication and result handling.
   
   ### Delivery and acceptance
   1. Hook + adapter: use Databricks connections and the actual common.ai BaseManagedAgentToolset extension contract. Expose space/conversation/message identity and structured results/errors; make common.ai optional. Document when generic Databricks MCP is sufficient and what the native API adds.
   2. Standalone operator: persist remote conversation/message IDs before downstream publication; reconnect to known operations after retry. Use asynchronous trigger polling and fetch the result on worker resume.
   3. Before promising safe submission retry, verify the remote API's idempotency/lookup guarantees. A worker can fail after server acceptance but before saving the ID. If recovery cannot disambiguate this window, report the limitation and require explicit recovery rather than silently creating another conversation.
   4. Cover timeout, access denial, remote failure, result pagination/size limits and trigger restart. Keep credentials and large result sets out of serialized triggers/XCom; return bounded metadata or artifact references.
   5. Preserve explicit user-supplied conversation ownership; cancellation must not interrupt unrelated work.
   
   ### Related issues
   This is a provider-local proposal, independent of #74191. Reuse existing resumable-job facilities only for Airflow versions that support them; document the supported matrix. Initial deliverable does not include Genie administration APIs or an Airflow HITL bridge.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Databricks: support Genie consultations and resumable standalone tasks [airflow]

["zozo123 (via GitHub)" <gi...@apache.org>]

zozo123 commented on issue #74194:
URL: https://github.com/apache/airflow/issues/74194#issuecomment-6073379181

   Thanks @garvit-arora, that slice is yours. Hook + managed-agent toolset adapter first, resumable standalone tasks later, exactly as scoped.
   
   A few things that will make review quick:
   - Build the adapter on the public `BaseManagedAgentToolset` contract from #73532 (`common/ai/toolsets/managed_agent.py`) and don't override private common.ai internals. Keep the adapter thin and separate from the hook, so the hook stands on its own and common.ai stays an optional extra.
   - Starting a conversation or posting a message is not idempotent. On an ambiguous failure (timeout, 5xx after send) surface an error rather than retrying, and only retry reads (get message, fetch result).
   - Bound result size and keep tokens out of anything serialized. Return space/conversation/message IDs plus a truncated, structured result.
   - Map 401/403/404/429 to clear errors. Note that Databricks sometimes answers a missing resource with 403.
   - Docs: say when the managed Genie MCP server via Unity Gateway is enough, and what the native API adds.
   - Please include one real-workspace run (PAT is fine) recorded separately from the mocked tests.
   
   My review on #74198 covers several of the same pitfalls if you want a checklist. Happy to review once it's up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Databricks: support Genie consultations and resumable standalone tasks [airflow]

["garvit-arora (via GitHub)" <gi...@apache.org>]

garvit-arora commented on issue #74194:
URL: https://github.com/apache/airflow/issues/74194#issuecomment-6067975002

   I'd like to take the first bounded slice: the Databricks Genie hook plus common-ai managed-agent toolset adapter, with connection auth, space/conversation/message identity, structured results/errors, optional common-ai dependency handling, docs, and mocked unit coverage. I will keep standalone resumable tasks out of this first PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org