You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pauldouane (via GitHub)" <gi...@apache.org> on 2025/09/01 08:54:57 UTC
[I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
pauldouane opened a new issue, #55128:
URL: https://github.com/apache/airflow/issues/55128
### Description
Add a feature to the DatabricksSqlOperator that allows direct export of query results to a Google Cloud Storage (GCS) bucket in Parquet and Avro formats.
### Use case/motivation
The current DatabricksSqlOperator allows executing SQL queries and saving the results to a file using the output_path and output_format parameters. However, it has a few limitations for common data engineering workflows:
It only supports csv, json, and jsonl formats. Parquet and Avro are widely used for their performance and schema handling.
The output is saved to the Databricks cluster's local filesystem (/tmp/), not directly to an object storage like GCS. This requires an additional step (a separate Airflow task, or in-Databricks logic) to move the file to its final destination, adding unnecessary complexity to the DAG.
This feature would streamline simple ETL/ELT pipelines by allowing a single Airflow task to:
Execute a SQL query on a Databricks warehouse.
Export the result as a Parquet or Avro file.
Save the file directly to a specified GCS path.
This would eliminate the need for an intermediate COPY command or a separate Databricks job (DatabricksSubmitRunOperator) for simple export scenarios, simplifying DAGs and improving readability.
Proposed Change:
Introduce new output_format values: Add support for 'parquet' and 'avro' to the output_format parameter.
Enhance output_path to support object storage URIs: The output_path parameter should be able to accept object storage URIs (e.g., gs://my-bucket/path/to/data.parquet).
Implement the export logic: The operator's internal logic would need to be updated to handle the conversion of the SQL query results to the specified format and stream them directly to the GCS location using the appropriate Databricks Spark APIs (e.g., spark.read.sql(...).write.format("parquet").save(...)).
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3242809728
Sure. Possibly somoene will pick it. Marked it as good-first-issue - > but if you would like to work on it yourself @pauldouane , feel free. This is the best to make sure that such a feature is added, otherwise someone will need to volunteer to implement it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3241488674
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "SaloniiSharmaa (via GitHub)" <gi...@apache.org>.
SaloniiSharmaa commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3263890041
Since this feature depends on Databricks’ native Spark APIs for Parquet and Avro, it makes sense to keep it in DatabricksSqlOperator rather than the generic SQL operator. I can work on a patch to extend output_format and support GCS paths properly. Any thoughts on making sure it fits cleanly with the current operator structure ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3251025650
> Ok in this case I will try to develop it, who should I put as reviewer?
You can't put anyone as reviewer. just open a PR and wait for reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3251028859
> I think this would be better put in the `SQLExecuteQueryOperator`, which `DatabricksSqlOperator` inherits. Thoughts?
I think this one is really about using native Databricks feature, not something "generic" for all databases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "pauldouane (via GitHub)" <gi...@apache.org>.
pauldouane commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3243239706
Ok in this case I will try the developer, who should I put in reviewer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 closed issue #55128: Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support
URL: https://github.com/apache/airflow/issues/55128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3245601425
I think this would be better put in the `SQLExecuteQueryOperator`, which `DatabricksSqlOperator` inherits. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add direct GCS export to DatabricksSqlOperator with Parquet/Avro support [airflow]
Posted by "akashadsare (via GitHub)" <gi...@apache.org>.
akashadsare commented on issue #55128:
URL: https://github.com/apache/airflow/issues/55128#issuecomment-3398313524
I would like to work on this issue.Please assign this to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org