You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/15 09:47:06 UTC
[GitHub] [airflow] saigopal opened a new pull request #13083: Update app.py
saigopal opened a new pull request #13083:
URL: https://github.com/apache/airflow/pull/13083
This change is required when airflow is running on openshift platform nodes enabled with system level FIPS. Without this airflow webserver won't start at all. The flask_caching module default hash method is md5 which is not allowed in FIPS mode, however it allows you to override the hash method.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745360560
> Yup, I understand. There is flag introduced in hashlib to deal with such scenarios where md5 is used for non-security related purposes.
>
> > > > import hashlib
> > > > **>>> hashlib.md5(usedforsecurity=False)**
> > > > <md5 HASH object @ 0x7f3bc1b9fed0>
> > > > hashlib.md5()
> > > > Traceback (most recent call last):
> > > > File "", line 1, in
> > > > ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
Oh? I can't see anything about this in the docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-787209747
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745244929
Well that's silly/stupid/annoying. By setting that system setting, it dsables accelerated/kernel provided md5.
But md5 is just used as a fingerprint mechanism. There will be many more cases than just this one --
```
airflow/cli/commands/webserver_command.py: hash_md5 = hashlib.md5()
airflow/cli/commands/webserver_command.py: hash_md5.update(chunk)
airflow/cli/commands/webserver_command.py: return hash_md5.hexdigest()
airflow/kubernetes/pod_generator.py: safe_hash = hashlib.md5(string.encode()).hexdigest()[:9]
airflow/kubernetes/pod_generator_deprecated.py: safe_hash = hashlib.md5(string.encode()).hexdigest()[:9]
airflow/models/serialized_dag.py: self.dag_hash = hashlib.md5(json.dumps(self.data, sort_keys=True).encode("utf-8")).hexdigest()
airflow/providers/google/cloud/hooks/bigquery.py: uniqueness_suffix = hashlib.md5(hash_base.encode()).hexdigest()
airflow/providers/google/cloud/hooks/gcs.py: def get_md5hash(self, bucket_name: str, object_name: str) -> str:
airflow/providers/google/cloud/hooks/gcs.py: blob_md5hash = blob.md5_hash
airflow/providers/google/cloud/hooks/gcs.py: self.log.info('The md5Hash of %s is %s', object_name, blob_md5hash)
airflow/providers/google/cloud/hooks/gcs.py: return blob_md5hash
airflow/providers/google/cloud/operators/bigquery.py: uniqueness_suffix = hashlib.md5(hash_base.encode()).hexdigest()
```
For instance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745181847
> The flask_caching module default hash method is md5 which is not allowed in FIPS mode.
[Citation Needed].
Saying "you can't use MD5 for anything" in FIPS is, quite frankly, hard to believe. This is never exposed to any user, nor is it a security measure. After all this usage is nothing to do with encryption.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] saigopal commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
saigopal commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745403216
I too got the "no keyword arguments" earlier. But then i tried like below and it worked.
>>> import hashlib
>>> h = hashlib.md5(usedforsecurity=False)
>>> h.update(b'test')
>>> h.hexdigest()
'098f6bcd4621d373cade4e832627b4f6'
>>>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745387027
> On the Internet, I found such a solution. Maybe that will be a better idea than dropping MD5.
>
> ```
> sed -i 's/hashlib.md5()/hashlib.md5(usedforsecurity=False)/g' <virtual_environment>/lib/python3.6/site-packages/django/db/backends/base/schema.py
> ```
Ace -- an undocumented parameter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] saigopal commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
saigopal commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745233658
The below error will appear when /proc/sys/crypto/fips_enabled is set to 1.
File "/home/airflow/.local/lib/python3.8/site-packages/flask_caching/backends/filesystemcache.py", line 149, in _get_filename
--
| hash = self._hash_method(key).hexdigest()
| ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745387027
> On the Internet, I found such a solution. Maybe that will be a better idea than dropping MD5.
>
> ```
> sed -i 's/hashlib.md5()/hashlib.md5(usedforsecurity=False)/g' <virtual_environment>/lib/python3.6/site-packages/django/db/backends/base/schema.py
> ```
Ace -- an undocumented parameter. That doesn't work on Python 3.7:
```
In [4]: hashlib.md5(b'abc', usedforsecurity=False)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-4-0fa59ad09572> in <module>
----> 1 hashlib.md5(b'abc', usedforsecurity=False)
TypeError: openssl_md5() takes no keyword arguments
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #13083:
URL: https://github.com/apache/airflow/pull/13083
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745181847
> The flask_caching module default hash method is md5 which is not allowed in FIPS mode.
[Citation Needed].
Saying "you can't use MD5 for anything" in FIPS is, quite frankly, hard to believe. This is never exposed to any user, nor is it a security measure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745175289
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] saigopal commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
saigopal commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-746124112
OS: Red Hat 8.3 on Openshift
Python 3.8.3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] saigopal commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
saigopal commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745291056
Yup, I understand. There is flag introduced in hashlib to deal with such scenarios where md5 is used for non-security related purposes.
>>> import hashlib
**>>> hashlib.md5(usedforsecurity=False)**
<md5 HASH object @ 0x7f3bc1b9fed0>
>>> hashlib.md5()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS
>>>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745387027
> On the Internet, I found such a solution. Maybe that will be a better idea than dropping MD5.
>
> ```
> sed -i 's/hashlib.md5()/hashlib.md5(usedforsecurity=False)/g' <virtual_environment>/lib/python3.6/site-packages/django/db/backends/base/schema.py
> ```
Ace -- an undocumented parameter. That doesn't work on Python 3.7:
```
In [4]: hashlib.md5(b'abc', usedforsecurity=False)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-4-0fa59ad09572> in <module>
----> 1 hashlib.md5(b'abc', usedforsecurity=False)
TypeError: openssl_md5() takes no keyword arguments
```
Looks like it's 3.9+ https://github.com/python/cpython/pull/20258
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745362651
On the Internet, I found such a solution. Maybe that will be a better idea than dropping MD5.
```
sed -i 's/hashlib.md5()/hashlib.md5(usedforsecurity=False)/g' <virtual_environment>/lib/python3.6/site-packages/django/db/backends/base/schema.py
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #13083: Update app.py
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #13083:
URL: https://github.com/apache/airflow/pull/13083#issuecomment-745966806
> I too got the "no keyword arguments" earlier. But then i tried like below and it worked.
>
> > > > import hashlib
> > > > h = hashlib.md5(usedforsecurity=False)
> > > > h.update(b'test')
> > > > h.hexdigest()
> > > > '098f6bcd4621d373cade4e832627b4f6'
What OS and what python did you test on? It's possible that someone has backported this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org