You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vargacypher (via GitHub)" <gi...@apache.org> on 2024/04/19 20:19:06 UTC
[I] SSHClickhouseOperator [airflow]
vargacypher opened a new issue, #39140:
URL: https://github.com/apache/airflow/issues/39140
### Description
Making calls to Clickhouse over ssh protocol could be make with SSHOperator or BashOperator, but could we abstract it more to make somenthing more specific to clickhouse ?
I don't know if it's make sense or just something that wee missed.
### Use case/motivation
Today we use a lot of ssh calls to connect to Clickhouse servers and execute commands there using **clickhouse-client** or use a intermediam server (ETL server) that has multiples Clikhouse server acess via Native Protocol, and executes the **clickhouse-client** commands from there using the Clickhouse Host.
I'd like to have and specific SSHClickhouseOperator and hook to do interect with clickhouse over ssh.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W closed issue #39140: SSHClickhouseOperator
URL: https://github.com/apache/airflow/issues/39140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "vargacypher (via GitHub)" <gi...@apache.org>.
vargacypher commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067672304
@eladkal sorry, I expressed myself poorly.
I will create the Clickhouse provider (hook and ClickhouseSSHOperator ), but I will not develop an operator based on its Python SDK as #25714 .
It's okay ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2084822512
Just followed through the discussion. Looks like we're going to have a separate Clickhouse provider not directly maintained in this repo. Thus, I think we're good to close this issue. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #39140: SSHClickhouseOperator
URL: https://github.com/apache/airflow/issues/39140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067261323
Thank for bringing up this idea. I propose that this would need to be added to any provider package. Even the SSHOperator is in a provider package (apache-airflow-providers-ssh) - For Clickhouse there is currently no provider package.
Starting a new provider package has some rules and needs a strong community to support. Be aware that "just dropping code" might be rejected or strongly challenged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "vargacypher (via GitHub)" <gi...@apache.org>.
vargacypher commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067270328
Thank u guys for the answer, so i would like to maintaining the provider if it's possible.
Do i need to follow additional steps besides the PR normal ???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067261562
This would likely require a new [community maintained provider](https://airflow.apache.org/docs/apache-airflow-providers/#community-maintained-providers), which would probably require a discussion in the [Airflow dev list](https://airflow.apache.org/community/) as to who would actually maintain such a provider.
If you're interested in maintaining the provider, then sure, this might be a great idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "vargacypher (via GitHub)" <gi...@apache.org>.
vargacypher commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067656156
O thanks @eladkal , but my approach it's a little bit different from the #25714 , since i pretend to make an derived SSHOperator.
Could i open a new PR since #25714 is closed ?
From this PR i see that i could use the hook, but i don't want to deal with that ClickhouseOperator at this moment, maybe in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067294313
Probably take a bit of reading in here: https://github.com/apache/airflow/blob/main/contributing-docs/11_provider_packages.rst#developing-community-managed-provider-packages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067636557
There was an attepmt to add ClickHouse provider
https://github.com/apache/airflow/issues/10893#issuecomment-1517683079
Anyone can pick it up from where it left off and follow the steps needed to add a new offical provider. I'm closing this issue as there is no community task here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SSHClickhouseOperator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39140:
URL: https://github.com/apache/airflow/issues/39140#issuecomment-2067666178
> O thanks @eladkal , but my approach it's a little bit different from the #25714 , since i pretend to make an derived SSHOperator. Could i open a new PR since #25714 is closed ? From this PR i see that i could use the hook, but i don't want to deal with that ClickhouseOperator at this moment, maybe in the future.
Feel free to submit PR directly.
I dont see how you can make this work without having ClickHouse hook (which requires ClickHouse provider) but if you have a way then simply submit a PR and we can review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org