You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alexio215 (via GitHub)" <gi...@apache.org> on 2025/01/24 18:10:07 UTC
[I] HIVE SSL mTLS capability [airflow]
alexio215 opened a new issue, #46023:
URL: https://github.com/apache/airflow/issues/46023
### Description
Add HIVE provider capability to use SSL and perform mTLS handshake with connecting host
### Use case/motivation
Airflow runs in a different network from HIVE due to policy and company support.
Require mTLS encrypted connection between the two instances to securely run HIVE jobs remotely.
### Related issues
None that I am aware of
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2637612311
I think I understand what you want to do, the way I see it there are two options.
1. Open a PR and if it's something simple and relevant it will also be promoted in the open source (I would love to do a CR for you).
2. Implement your own wrapper for the operator in your organization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2613116453
Hello, thank you for having me. It is my first time contributing to any open source project, so please bear with me.
Happy to learn from any wisdom shared
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #46023: HIVE SSL mTLS capability
URL: https://github.com/apache/airflow/issues/46023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2626422301
So I'm currently looking at using two capabilities. The first, is to connect to an NGINX proxy that requires SSL certs and expects mTLS to serve HIVE commands locally through our cluster, into the HIVE2SERVER running right behind it. The second, down the line that I am hoping for, is to find or create support for direct connection with pyHIVE to HIVE2SERVER running with SSL, and to perform mTLS. The problem with this however is that I notice that python does not natively support the .jks format that HIVE2SERVER expects, hence the use of an NGINX proxy. However, looking at pyHIVE, and its most recent issues, to me it seems that pyHIVE as well does not support SSL connection:
https://github.com/dropbox/PyHive/issues/257
Forgive me for any misunderstanding as well, this is all a learning process to me at the same time. Thank you for the patience and help @nevcohen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2632478556
> > > So I'm currently looking at using two capabilities. The first, is to connect to an NGINX proxy that requires SSL certs and expects mTLS to serve HIVE commands locally through our cluster, into the HIVE2SERVER running right behind it. The second, down the line that I am hoping for, is to find or create support for direct connection with pyHIVE to HIVE2SERVER running with SSL, and to perform mTLS. The problem with this however is that I notice that python does not natively support the .jks format that HIVE2SERVER expects, hence the use of an NGINX proxy. However, looking at pyHIVE, and its most recent issues, to me it seems that pyHIVE as well does not support SSL connection:
> > > [dropbox/PyHive#257](https://github.com/dropbox/PyHive/issues/257)
> > > Forgive me for any misunderstanding as well, this is all a learning process to me at the same time. Thank you for the patience and help [@nevcohen](https://github.com/nevcohen)
> >
> >
> > So today how do you connect to hive using a code?
>
> Thank you for the patience, this has taken some digging on my end, getting accustomed to what is currently practiced in my org. Currently our pyHive queries are written a more manual script and sent to a NGINX server that redirects appropriate traffic to a Hive2Server proxy. The Thrift communication is wrapped in HTTPS using the THTTPClient module from the Thrift library. I have found this to exist within pyHive as well.
>
> This lives and is made accessible within the Connection method of pyHive `if scheme in ("https", "http") and thrift_transport is None: port = port or 1000 ssl_context = None if scheme == "https": ssl_context = create_default_context() ssl_context.check_hostname = check_hostname == "true" ssl_cert = ssl_cert or "none" ssl_context.verify_mode = ssl_cert_parameter_map.get(ssl_cert, CERT_NONE) thrift_transport = thrift.transport.THttpClient.THttpClient( uri_or_host="{scheme}://{host}:{port}/cliservice/".format( scheme=scheme, host=host, port=port ), ssl_context=ssl_context, )`
>
> My goal is to add a method using the ssl library that creates ssl context using the extras provided and appends them to the connection being created if a "use_https_proxy" boolean is specified within the proxy. Further, a "enable_mtls" boolean option will be included to allow for cases where someone needs to use mTLS.
Finding a way to do this through the pyHive scheme and default constructor parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2646644945
> I think I understand what you want to do, the way I see it there are two options.
>
> 1. Open a PR and if it's something simple and relevant it will also be promoted in the open source (I would love to do a CR for you).
> 2. Implement your own wrapper for the operator in your organization.
Hello, thank you for the help. I have opened an issue to make a PR for puHive first since it is lacking the capability fundamentally. Once I get that merged, I will come back here to make a PR for the airflow provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2632079316
> > So I'm currently looking at using two capabilities. The first, is to connect to an NGINX proxy that requires SSL certs and expects mTLS to serve HIVE commands locally through our cluster, into the HIVE2SERVER running right behind it. The second, down the line that I am hoping for, is to find or create support for direct connection with pyHIVE to HIVE2SERVER running with SSL, and to perform mTLS. The problem with this however is that I notice that python does not natively support the .jks format that HIVE2SERVER expects, hence the use of an NGINX proxy. However, looking at pyHIVE, and its most recent issues, to me it seems that pyHIVE as well does not support SSL connection:
> > [dropbox/PyHive#257](https://github.com/dropbox/PyHive/issues/257)
> > Forgive me for any misunderstanding as well, this is all a learning process to me at the same time. Thank you for the patience and help [@nevcohen](https://github.com/nevcohen)
>
> So today how do you connect to hive using a code?
Thank you for the patience, this has taken some digging on my end, getting accustomed to what is currently practiced in my org. Currently our pyHive queries are written a more manual script and sent to a NGINX server that redirects appropriate traffic to a Hive2Server proxy. The Thrift communication is wrapped in HTTPS using the THTTPClient module from the Thrift library. I have found this to exist within pyHive as well.
This lives and is made accessible within the Connection method of pyHive
`if scheme in ("https", "http") and thrift_transport is None:
port = port or 1000
ssl_context = None
if scheme == "https":
ssl_context = create_default_context()
ssl_context.check_hostname = check_hostname == "true"
ssl_cert = ssl_cert or "none"
ssl_context.verify_mode = ssl_cert_parameter_map.get(ssl_cert, CERT_NONE)
thrift_transport = thrift.transport.THttpClient.THttpClient(
uri_or_host="{scheme}://{host}:{port}/cliservice/".format(
scheme=scheme, host=host, port=port
),
ssl_context=ssl_context,
)`
My goal is to add a method using the ssl library that creates ssl context using the extras provided and appends them to the connection being created if a "use_https_proxy" boolean is specified within the proxy. Further, a "enable_mtls" boolean option will be included to allow for cases where someone needs to use mTLS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2622767231
The `hive cli` or `hive server 2` [connections](https://airflow.apache.org/docs/apache-airflow-providers-apache-hive/stable/connections/index.html) don't work for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2613128436
My current inclination in solving this problem is to use [jpype](https://pypi.org/project/jpype1/) to funnel python requests through a JVM running the JDBC Driver adjacent to Airflow.
The goal with this is to use the Python natively to write HIVE DAGs but communicate in Java which is more native to HIVE and supports the JKS key format, which is the default to HIVE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2613250678
On pause for now and considering ramifications of this in my environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2613115204
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2629484716
> So I'm currently looking at using two capabilities. The first, is to connect to an NGINX proxy that requires SSL certs and expects mTLS to serve HIVE commands locally through our cluster, into the HIVE2SERVER running right behind it. The second, down the line that I am hoping for, is to find or create support for direct connection with pyHIVE to HIVE2SERVER running with SSL, and to perform mTLS. The problem with this however is that I notice that python does not natively support the .jks format that HIVE2SERVER expects, hence the use of an NGINX proxy. However, looking at pyHIVE, and its most recent issues, to me it seems that pyHIVE as well does not support SSL connection:
> https://github.com/dropbox/PyHive/issues/257
>
> Forgive me for any misunderstanding as well, this is all a learning process to me at the same time. Thank you for the patience and help @nevcohen
So today how do you connect to hive using a code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "alexio215 (via GitHub)" <gi...@apache.org>.
alexio215 commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2695338668
Just wanted to add the comment, that the new pyHive has been adopted by the apache/kyuubi project. The PR for this support has been made upstream, and is awaiting release.
Issue can remain closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HIVE SSL mTLS capability [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #46023:
URL: https://github.com/apache/airflow/issues/46023#issuecomment-2692638260
I'm closing this issue as it's missing feature in upstream library https://github.com/dropbox/PyHive/issues/480
Should upstream add support for it feel free to open PR directly (no need for issue in Airflow)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org