You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Mrhacktivist (via GitHub)" <gi...@apache.org> on 2024/02/07 16:52:56 UTC
[I] Airflow CLI command "dags list" command returning unexpected output [airflow]
Mrhacktivist opened a new issue, #37228:
URL: https://github.com/apache/airflow/issues/37228
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.5.1
### What happened?
Airflow CLI command "dags list" command returning unexpected output containing "Sample operation result: <Task(DatabricksSqlOperator): select_data>
Sample operation type: <class 'airflow.providers.databricks.operators.databricks_sql.DatabricksSqlOperator'>" in its response
### What you think should happen instead?
_No response_
### How to reproduce
It's getting returned in specific environment only. Tried in other environments with same version and there is no such output
### Operating System
amazon linux 2
### Versions of Apache Airflow Providers
_No response_
### Deployment
Amazon (AWS) MWAA
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow CLI command "dags list" command returning unexpected output [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37228:
URL: https://github.com/apache/airflow/issues/37228#issuecomment-1934087465
Closing as invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow CLI command "dags list" command returning unexpected output [airflow]
Posted by "Mrhacktivist (via GitHub)" <gi...@apache.org>.
Mrhacktivist commented on issue #37228:
URL: https://github.com/apache/airflow/issues/37228#issuecomment-1933855751
CLI USED -
```python
```
import boto3
import json
import requests
import base64
mwaa_env_name = 'MyAirflowEnvironment5'
mwaa_cli_command = 'dags list -o json'
client = boto3.client('mwaa')
mwaa_cli_token = client.create_cli_token(
Name=mwaa_env_name
)
mwaa_auth_token = 'Bearer ' + mwaa_cli_token['CliToken']
mwaa_webserver_hostname = 'https://{0}/aws_mwaa/cli'.format(mwaa_cli_token['WebServerHostname'])
raw_data = '{0}'.format(mwaa_cli_command)
mwaa_response = requests.post(
mwaa_webserver_hostname,
headers={
'Authorization': mwaa_auth_token,
'Content-Type': 'text/plain'
},
data=raw_data
)
mwaa_std_err_message = base64.b64decode(mwaa_response.json()['stderr']).decode('utf8')
mwaa_std_out_message = base64.b64decode(mwaa_response.json()['stdout']).decode('utf8')
print(mwaa_response.status_code)
print(mwaa_std_err_message)
print(mwaa_std_out_message)
```
```
Output -
```
Sample operation result: <Task(DatabricksSqlOperator): select_data>
Sample operation type: <class 'airflow.providers.databricks.operators.databricks_sql.DatabricksSqlOperator'>
[{"dag_id": "pl_CustomerPortfolios_Load", "filepath": "pl_CustomerPortfolios_Load.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_api", "filepath": "pl_api.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_combined_ingestion", "filepath": "pl_combined_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_copy_source_file_from_ingress", "filepath": "pl_copy_source_file_from_ingress.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_database_cdc_full_ingestion", "filepath": "pl_database_cdc_full_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_database_cdc_ingestion", "filepath": "pl_database_cdc_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_database_full_ingestion", "filepath": "pl_database_full_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_database_ingestion", "filepath": "pl_database_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_database_inge
stion_hotelkey", "filepath": "pl_database_ingestion_hotelkey.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_database_ingestion_onprem", "filepath": "pl_database_ingestion_onprem.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_email_notification", "filepath": "pl_email_notification.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_file_ingestion", "filepath": "pl_file_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_initiate_ingestion", "filepath": "pl_initiate_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_int021_Load", "filepath": "pl_int021_Load.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_invoke_databricks_workflow", "filepath": "pl_invoke_databricks_workflow.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_oci_ingestion", "filepath": "pl_oci_ingestion.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_onyx_to_egress", "filepath": "pl_onyx_to_egress.py", "owner": "air
flow", "paused": "False"}, {"dag_id": "pl_prep_layer_only", "filepath": "pl_prep_layer_only.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_prep_only", "filepath": "pl_prep_only.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_pyspark_test", "filepath": "pl_pyspark_test.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_raw_ingestion_log", "filepath": "pl_raw_ingestion_log.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_snowflake", "filepath": "pl_snowflake.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_source_table_count_debug", "filepath": "pl_source_table_count_debug.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_tableau_trigger", "filepath": "pl_tableau_trigger.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_test_airflow_s3_connections", "filepath": "pl_test_airflow_s3_connections.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_test_db_connectivity", "filepath": "pl_test_db_connectivity
.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_test_exception", "filepath": "pl_test_exception.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_test_ingress_connection", "filepath": "pl_test_ingress_connection.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_test_new_ingress_connection", "filepath": "pl_test_new_ingress_connection.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_threshold_trigger", "filepath": "pl_threshold_trigger.py", "owner": "airflow", "paused": "False"}, {"dag_id": "pl_zip_files", "filepath": "pl_zip_files.py", "owner": "airflow", "paused": "False"}]
```
Yes the respective operator is used in DAG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow CLI command "dags list" command returning unexpected output [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #37228:
URL: https://github.com/apache/airflow/issues/37228#issuecomment-1932975964
Can you please post some more details about the unexpected details?
Can you please post the full CLI you used?
Also if possible the full output would be good. Redacted as well.
Can you please tell us whether the reported operators are used in your DAGs (assuming yes)?
Can you check whether the same is happening on the most recent version of Airflow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow CLI command "dags list" command returning unexpected output [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37228: Airflow CLI command "dags list" command returning unexpected output
URL: https://github.com/apache/airflow/issues/37228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow CLI command "dags list" command returning unexpected output [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37228:
URL: https://github.com/apache/airflow/issues/37228#issuecomment-1934086764
Well, those messages come from your DAG or configuraiton (local settings) while the dags are being parsed. Just don't print output to stdout in your DAGs when they are parsed.
Look for the places where your DAGs write those messages. The problem is that parsing DAG executes your code so if you do something in your code to use stderr, this creates the problem.
We currently suppress logs and warnings for all commands that return any json output with decorator below - but it does not suppress stuff that you print manually in your dags using `print()`
Generally it's a good practice to use logging facilities to print diagnostics info rather than printing to stdout directly.
```
def suppress_logs_and_warning(f: T) -> T:
"""Suppress logging and warning messages in cli functions."""
@functools.wraps(f)
def _wrapper(*args, **kwargs):
_check_cli_args(args)
if args[0].verbose:
f(*args, **kwargs)
else:
with warnings.catch_warnings():
warnings.simplefilter("ignore")
logging.disable(logging.CRITICAL)
try:
f(*args, **kwargs)
finally:
# logging output again depends on the effective
# levels of individual loggers
logging.disable(logging.NOTSET)
return cast(T, _wrapper)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow CLI command "dags list" command returning unexpected output [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37228:
URL: https://github.com/apache/airflow/issues/37228#issuecomment-1932473829
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org