You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Rurushu13 (via GitHub)" <gi...@apache.org> on 2024/01/09 11:49:21 UTC

[I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Rurushu13 opened a new issue, #36688:
URL: https://github.com/apache/airflow/issues/36688

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.5.1
   
   ### What happened?
   
   While we were trying to use a BashOperator that would take our Database URI as an Env variable, we noticed that one of our connection URI's password aren't correctly being masked. The difference we found out between two passwords are, one of them have no special character whereas the other one has exclamation points. Original passphrase is `!Weird!Password^123$`
   ![image](https://github.com/apache/airflow/assets/110015112/e59ee64f-bc7b-4095-8a46-16a9c12f74cf)
   
   In order to confirm that this was indeed the issue, we tried removing any special characters from the password of the errenous one, and it was indeed the issue. Password ended up being masked.
   ![image](https://github.com/apache/airflow/assets/110015112/f0273f5c-4b93-4a34-8bca-75d26307b5b0)
   
   
   
   
   ### What you think should happen instead?
   
   The password should be masked even if it has a special character in it.
   
   ### How to reproduce
   
   1. Create a BashOperator (honestly I think it would work with any operator)
   2. Pass a Connection URI of a database that has special characters (we had issue with `!`) in `env` parameter using `BaseHook.get_connection("Database").get_uri()`
   3. Check rendered template of the operator in Airflow UI
   
   ### Operating System
   
   Ventura 13.5
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   It occurs every-time.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #36688: Airflow does not mask secrets in Database URI's when secret has a special character.
URL: https://github.com/apache/airflow/issues/36688


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "Rurushu13 (via GitHub)" <gi...@apache.org>.
Rurushu13 commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1883340047

   > @Rurushu13
   > Good finding. If possible, can you share the sample DAG?
   
   @dirrao There it is
   
   `from datetime import timedelta
   import pendulum
   import os
   
   from airflow.decorators import dag
   from airflow.operators.bash import BashOperator
   from airflow.hooks.base import BaseHook
   
   
   @dag(
       dag_id="shell_test",
       description='testing the shell script',
       start_date=pendulum.datetime(2023, 1, 1, tz="Europe/Istanbul"),
       schedule='@daily',
       catchup=False,
       tags=['test'],
       default_args={
           "retries": 0,
           "retry_delay": timedelta(minutes=5),
       },
       template_searchpath=['include/sql/datawarehouse/source',
                            'include/sql/datawarehouse/stage',
                            'include/sql/datawarehouse/final'],)
   def shell_test():
       airflow_home = os.environ.get('AIRFLOW_HOME')
       shell_test_task = BashOperator(
           task_id="shell_test_task",
           bash_command=f"/bin/bash {airflow_home}/include/scripts/datawarehouse/transfer_script.sh ",
           env={
                **{'TARGET_CONNECTION_STRING': BaseHook.get_connection("DATAWAREHOUSE").get_uri(),
                   'SOURCE_CONNECTION_STRING': BaseHook.get_connection("EFABRIKA").get_uri()},
                }
       )
   
   
   shell_test()`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1901922955

   I think @aritra24 is right - this is not about URL passwords (where the password should - indeed - be URL, encoded) - but if there is a password in connection, that does not get masked when we print a URL form of the connnection, then the quoted version should be masked.
   
   While we cannot prevent all the ways how password can be mangled and transformed (say you add space and Base64 such password for example), some common usages (like URL encoding) that are likely to appear in our logs should be masked as welll. If I read this one properly, the connection password is stored "as is" while we are quoting it i the output when printing it as "url" so we should mask it there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1884429546

   @dirrao yes, the main branch also has this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "Rurushu13 (via GitHub)" <gi...@apache.org>.
Rurushu13 commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1914623055

   Thank you for your hard work everyone 🤗 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1883390997

   I was able to replicate this, seems like a valid bug. Looking into it. 🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1900060730

   @amoghrajesh while that might be, but the issue is that the password when encoded is not getting redacted (Because our redactor today explicitly redacts exact matches only). I do not believe they're asking that it not be encoded, but that on encoding it continue to get redacted like a alphanumeric password would and that's what the PR I've raised does. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1883324665

   @Rurushu13 
   Good finding. If possible, can you share the sample DAG? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1884158050

   > I was able to replicate this, seems like a valid bug. Looking into it. 🤔
   
   is this happening on the airflow main branch? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1899914323

   I think it is by design that we have to percent encode the passwords containing special characters. The postgres documentation also has a mention of it: https://www.postgresql.org/docs/11/libpq-connect.html#id-1.7.3.8.3.6
   
   Also check: https://dba.stackexchange.com/questions/243219/in-postgresql-url-i-cant-use-a-password-containing-special-characters
   
   I think what you need to do here is define the env variables post encoding, maybe that should solve the issue.
   
   cc @uranusjr, I think you can have some valuable suggestions here cos of the new AIP coming in from you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Airflow does not mask secrets in Database URI's when secret has a special character. [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #36688:
URL: https://github.com/apache/airflow/issues/36688#issuecomment-1902520339

   Thank you for the explanations, understood, @potiuk and @aritra24. Looking at the PR now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org