You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/01 19:16:21 UTC
[GitHub] [airflow] turbaszek opened a new pull request #14546: Add plain format output to cli tables
turbaszek opened a new pull request #14546:
URL: https://github.com/apache/airflow/pull/14546
closes: #14517
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788767906
There may be other situations where WARNING may jump out. Shall we use the decorator `utils.cli.suppress_logs_and_warning` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788500772
> In the original implementation that @mik-laj mentioned (#8409), header is available ("DAG ID", "Filepath", "Owner"), but they seem not available here.
>
> May be good to have them?
But this will require using `grep -v` to remove the header, if the purpose is to use it in pipes I think it's better to not have it. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on a change in pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on a change in pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#discussion_r585016991
##########
File path: airflow/cli/simple_table.py
##########
@@ -62,6 +63,14 @@ def print_as_table(self, data: List[Dict]):
table.add_row(*[str(d) for d in row.values()])
self.print(table)
+ def print_as_plain_table(self, data: List[Dict]):
+ """Renders list of dictionaries as a simple table than can be easily piped"""
+ if not data:
+ self.print("No data found")
+ return
+ output = tabulate(data, tablefmt="plain")
Review comment:
```suggestion
output = tabulate(data, tablefmt="plain", headers='keys')
```
This change should do what I mentioned above. Referenced from the original implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788929886
> There may be other situations where WARNING may jump out. Shall we use the decorator `utils.cli.suppress_logs_and_warning` here?
The decorator is already there, I'm afraid that the warning is raised before initialising airflow - but that's something for another PR I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788682694
> > In the original implementation that @mik-laj mentioned (#8409), header is available ("DAG ID", "Filepath", "Owner"), but they seem not available here.
> > May be good to have them?
>
> But this will require using `grep -v` to remove the header, if the purpose is to use it in pipes I think it's better to not have it. WDYT?
I agree it’s a valid concern.
The main reason for what I propose this is that as a dumb person I may need to know what’s exactly the a few columns given to me (it’s obvious what the values’ lookings of course, but may be good to be more explicit).
a possible solution may be to describe in the doc/help msg what are the columns printed if “plain” output is chosen?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788957189
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788744129
Done
```
root@e5dd824b21f3:/opt/airflow# airflow dags list -o plain
/opt/airflow/airflow/providers/apache/beam/hooks/beam.py:28 DeprecationWarning: This module is deprecated. Please use `airflow.hooks.base`.
[2021-03-02 08:58:39,379] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_type attribute and cannot be registered
[2021-03-02 08:58:39,379] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_name_attr attribute and cannot be registered
[2021-03-02 08:58:39,380] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing hook_name attribute and cannot be registered
[2021-03-02 08:58:42,256] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_type attribute and cannot be registered
[2021-03-02 08:58:42,256] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_name_attr attribute and cannot be registered
[2021-03-02 08:58:42,256] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing hook_name attribute and cannot be registered
dag_id filepath owner paused
example_bash_operator /opt/airflow/airflow/example_dags/example_bash_operator.py airflow True
example_branch_dop_operator_v3 /opt/airflow/airflow/example_dags/example_branch_python_dop_operator_3.py airflow True
example_branch_operator /opt/airflow/airflow/example_dags/example_branch_operator.py airflow True
example_complex /opt/airflow/airflow/example_dags/example_complex.py airflow True
example_dag_decorator /opt/airflow/airflow/example_dags/example_dag_decorator.py airflow True
example_external_task_marker_child /opt/airflow/airflow/example_dags/example_external_task_marker_dag.py airflow True
example_external_task_marker_parent /opt/airflow/airflow/example_dags/example_external_task_marker_dag.py airflow True
example_kubernetes_executor /opt/airflow/airflow/example_dags/example_kubernetes_executor.py airflow True
example_kubernetes_executor_config /opt/airflow/airflow/example_dags/example_kubernetes_executor_config.py airflow True
example_nested_branch_dag /opt/airflow/airflow/example_dags/example_nested_branch_dag.py airflow True
example_passing_params_via_test_command /opt/airflow/airflow/example_dags/example_passing_params_via_test_command.py airflow True
example_python_operator /opt/airflow/airflow/example_dags/example_python_operator.py airflow True
example_short_circuit_operator /opt/airflow/airflow/example_dags/example_short_circuit_operator.py airflow True
example_skip_dag /opt/airflow/airflow/example_dags/example_skip_dag.py airflow True
example_subdag_operator /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow True
example_subdag_operator.section-1 /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow True
example_subdag_operator.section-2 /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow True
example_task_group /opt/airflow/airflow/example_dags/example_task_group.py airflow True
example_trigger_controller_dag /opt/airflow/airflow/example_dags/example_trigger_controller_dag.py airflow True
example_trigger_target_dag /opt/airflow/airflow/example_dags/example_trigger_target_dag.py airflow True
example_weekday_branch_operator /opt/airflow/airflow/example_dags/example_branch_day_of_week_operator.py airflow True
example_xcom /opt/airflow/airflow/example_dags/example_xcom.py airflow True
example_xcom_args /opt/airflow/airflow/example_dags/example_xcomargs.py airflow True
example_xcom_args_with_operators /opt/airflow/airflow/example_dags/example_xcomargs.py airflow True
latest_only /opt/airflow/airflow/example_dags/example_latest_only.py airflow True
latest_only_with_trigger /opt/airflow/airflow/example_dags/example_latest_only_with_trigger.py airflow True
test_utils /opt/airflow/airflow/example_dags/test_utils.py airflow True
tutorial /opt/airflow/airflow/example_dags/tutorial.py airflow True
tutorial_etl_dag /opt/airflow/airflow/example_dags/tutorial_etl_dag.py airflow True
tutorial_taskflow_api_etl /opt/airflow/airflow/example_dags/tutorial_taskflow_api_etl.py airflow True
```
There are some nasty warning but I think that #14554 is taking care of that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788344561
Should we update docs? http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/usage-cli.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788777053
> Should we update docs? http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/usage-cli.html
WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788951475
> > There may be other situations where WARNING may jump out. Shall we use the decorator `utils.cli.suppress_logs_and_warning` here?
>
> The decorator is already there, I'm afraid that the warning is raised before initialising airflow - but that's something for another PR I think.
I see. Then all good. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek merged pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
turbaszek merged pull request #14546:
URL: https://github.com/apache/airflow/pull/14546
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788688568
We want to add this format for the best compatibility with various word processing tools. I have given AWK as an example, but other people can parse these tables with other tools. For this reason, it seems to me that we should imitate the behavior of other tools.
Docker display headers
```
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a6b1b5ce7137 postgres:13 "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 5432/tcp airflow_postgres_1
d4dddf3b3b10 redis:latest "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 0.0.0.0:6379->6379/tcp airflow_redis_1
```
gcloud display headers
```
PROJECT_ID NAME PROJECT_NUMBER
airflow-spy airflow-spy REDACTED
airflow-triage-party airflow-triage-party REDACTED
anierobi-bastion anierobi-bastion REDACTED
```
So I believe that we should also display the header and not create another data representation format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] XD-DENG commented on pull request #14546: Add plain format output to cli tables
Posted by GitBox <gi...@apache.org>.
XD-DENG commented on pull request #14546:
URL: https://github.com/apache/airflow/pull/14546#issuecomment-788689977
> We want to add this format for the best compatibility with various word processing tools. I have given AWK as an example, but other people can parse these tables with other tools. For this reason, it seems to me that we should imitate the behavior of other tools.
> Docker display headers
>
> ```
> $ docker ps
> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> a6b1b5ce7137 postgres:13 "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 5432/tcp airflow_postgres_1
> d4dddf3b3b10 redis:latest "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 0.0.0.0:6379->6379/tcp airflow_redis_1
> ```
>
> gcloud display headers
>
> ```
> PROJECT_ID NAME PROJECT_NUMBER
> airflow-spy airflow-spy REDACTED
> airflow-triage-party airflow-triage-party REDACTED
> anierobi-bastion anierobi-bastion REDACTED
> ```
>
> So I believe that we should also display the header and not create another data representation format.
Thanks @mik-laj for the examples. Makes sense to me.
@turbaszek WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org