You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/10/09 22:38:44 UTC
[GitHub] [arrow-datafusion] rupurt opened a new issue #1096: Vision for datafusion-cli
rupurt opened a new issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096
Howdy,
I'm so pumped I found this project! Very excited to use more of it to improve the speed of my ETL pipelines. Thank you to the maintainers :)
Are there any resources for the vision of each component? Specifically it would be helpful to understand the vision for the datafusion-cli. In my head it feels like it would be natural to replicate as much `psql` functionality as possible but would like to understand your frame of mind as maintainers.
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Understand the vision of datafusion-cli so I can make useful contributions and improvements e.g. import from CSV and export to parquet via `datafusion-cli`
**Describe the solution you'd like**
Support for common psql commands like `\copy, \d` etc... instead of writing a custom rust program.
**Describe alternatives you've considered**
-
**Additional context**
-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist edited a comment on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
Jimexist edited a comment on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939418770
Thanks for raising this issue and question @rupurt.
First off, I don't believe there's any centralized view on what datafusion-cli is or isn't, and there's no process yet to determine that. However I can share some of my thinkings here.
## What datafusion-cli is not
Our ecosystem is full of different tools to manipulate data, each fitting its own niche purpose. In my opinion, datafusion-cli should not try to be yet another general-purpose tool to just manipulate data, especially since datafusion itself is intended to be an embeddable component for other tools (e.g. [cube](https://cube.dev/), [ballista](https://github.com/ballista-compute/ballista), [roapi](https://roapi.github.io/docs/)), to avoid confusion or reduce fragmented tech investment.
Specifically, datafusion-cli is not:
1. a general purpose Python enabled tool to query data, for that you'll have the [Python binding](https://pypi.org/project/datafusion) for datafusion itself or [polar](https://github.com/pola-rs/polars) for its speed and pandas compatibility, both leveraging Arrow and datafusion underneath
2. a command line tool to manipulate small, tabular or structured data, for that you'll have [xsv](https://github.com/BurntSushi/xsv), [jq](https://github.com/stedolan/jq), or [rq](https://github.com/dflemstr/rq), depending on the file formats that one wants
3. a client to an HTTP or GraphQL enabled server backend, for that, you can have [roapi](https://roapi.github.io/docs/) or similar things, or in many cases Spark or Presto is just fine (when data size is large)
Also for 3. please note that (AFAIK) datafusion and datafusion-cli themselves do not concern with distributed computing, i.e. data sharding is something built on top of them - they can only do in-memory, uniformly accessible data manipulation.
## What datafusion-cli can be
Given the above assertions, I believe the place where datafusion-cli can shine is:
1. the data size is large enough so that simple tools like `jq` or `xsv` can't cut it (within reasonable amount of time), but still small enough that can be fit into memory ([EC2 machine has up to 12TB](https://aws.amazon.com/about-aws/whats-new/2021/05/four-ec2-high-memory-instances-with-up-to-12tb-memory-available-with-on-demand-and-savings-plan-purchase-options/)) - if you do care about the speed
2. when there's no need or necessity to keep a long running server or adopt full stack of Spark or Presto cluster due to their high maintenance cost
I think in many cases of ETL you do encounter these type of need for “glue” script that can just run with a simple shell script, either triggered manually, via Airflow, or crontab, etc. And it runs to the end, without any setup needed, takes in one or more parquet files, an SQL script, and spits out target data, either in JSON, CSV, or maybe Parquet.
It can also be a REPL, an interactive shell to quickly verify commands and look into data
## Considerations on API
I think there are two aspects of API to this CLI:
1. the command line API (flags, options, naming, etc.)
2. the query syntax itself
For 2. it's clear that Postgres-compatible SQL is the choice here, but for 1. I don't think we are necessarily bound to the peculiarity of `psql` itself because:
1. it's in many cases specific to Postgres and the fact that it's a client-server architecture
2. `psql` is in many cases used interactively, but when it comes to shell automation, there are all sorts of other scripts in use, e.g. `pg_ctl`, `pg_dumpall`, etc. and here they all map to datafusion-cli
Having said that, I do think commands like `\copy` are already familiarised within users and nice to have, but I admit that in many cases I am still confused on the different flags and behaviors when that command is used interactively versus used as a CLI option. We need to be more consistent here, and just not necessarily consistent with `psql` itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939412777
@Jimexist did a lot of work around datafusion-cli. From what could tell, datafusion tries to be as close to postgres as possible, so it makes sense to me for us to support/match psql commands as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-951903912
I think this one is now done, so closing ticket -- would love to have your help @rupurt !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
Jimexist commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-953505440
related https://github.com/Homebrew/homebrew-core/pull/88184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist edited a comment on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
Jimexist edited a comment on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939418770
Thanks for raising this issue and question @rupurt.
First off, I don't believe there's any centralized view on what datafusion-cli is or isn't, and there's no process yet to determine that. However I can share some of my thinkings here.
## What datafusion-cli is not
Our ecosystem is full of different tools to manipulate data, each fitting its own niche purpose. In my opinion, datafusion-cli should not try to be yet another general-purpose tool to just manipulate data, especially since datafusion itself is intended to be an embeddable component for other tools (e.g. [cube](https://cube.dev/), [ballista](https://github.com/ballista-compute/ballista), [roapi](https://roapi.github.io/docs/)), to avoid confusion or reduce fragmented tech investment.
Specifically, datafusion-cli is not:
1. a general purpose Python enabled tool to query data, for that you'll have the [Python binding](https://pypi.org/project/datafusion) for datafusion itself or [polar](https://github.com/pola-rs/polars) for its speed and pandas compatibility, both leveraging Arrow and datafusion underneath
2. a command line tool to manipulate small, tabular or structured data, for that you'll have [xsv](https://github.com/BurntSushi/xsv), [jq](https://github.com/stedolan/jq), or [rq](https://github.com/dflemstr/rq), depending on the file formats that one wants
3. a client to an HTTP or GraphQL enabled server backend, for that, you can have [roapi](https://roapi.github.io/docs/) or similar things, or in many cases Spark or Presto is just fine (when data size is large)
Also for 3. please note that (AFAIK) datafusion and datafusion-cli themselves do not concern with distributed computing, i.e. data sharding is something built on top of them - they can only do in-memory, uniformly accessible data manipulation.
## What datafusion-cli can be
Given the above assertions, I believe the place where datafusion-cli can shine is:
1. the data size is large enough so that simple tools like `jq` or `xsv` can't cut it (within reasonable amount of time), but still small enough that can be fit into memory ([EC2 machine has up to 12TB](https://aws.amazon.com/about-aws/whats-new/2021/05/four-ec2-high-memory-instances-with-up-to-12tb-memory-available-with-on-demand-and-savings-plan-purchase-options/)) - if you do care about the speed
2. when there's no need or necessity to keep a long running server or adopt full stack of Spark or Presto cluster due to their high maintenance cost
I think in many cases of ETL you do encounter these type of need for “glue” script that can just run with a simple shell script, either triggered manually, via Airflow, or crontab, etc. And it runs to the end, without any setup needed, takes in one or more parquet files, an SQL script, and spits out target data, either in JSON, CSV, or maybe Parquet.
It can also be a REPL, an interactive shell to quickly verify commands and look into data
## Considerations on API
I think there are two aspects of API to this CLI:
1. the command line API (flags, options, naming, etc.)
2. the query syntax itself
For 2. it's clear that Postgres-compatible SQL is the choice here, but for 1. I don't think we are necessarily bound to the peculiarity of `psql` itself because:
1. it's in many cases specific to Postgres and the fact that it's a client-server architecture
2. `psql` is in many cases used interactively, but when it comes to shell automation, there are all sorts of other scripts in use, e.g. `pg_ctl`, `pg_dumpall`, etc. and here they all map to datafusion-cli
Having said that, I do think commands like `\copy` are already familiarised within users and nice to have, but I admit that in many cases I am still confused on the different flags and behaviors when that command is used interactively versus used as a CLI option. We need to be more consistent here, and just not necessarily consistent with `psql` itself.
## What comes next
Although I'm not recently working on this area, some so call "roadmap" that I have in mind would be:
1. to build a fully abstracted layer of repl parsing so that queries and commands are separated and handled correctly - currently it's kind of a hack (e.g. SIG_INT isn't properly handled)
2. hook up the stats subsystem and have the cli print out more stats for query debugging, etc.
3. better error handling for interactive use _and_ shell scripting usage
2. to widen the usage by publishing to apt, brew, and possible NuGet registry so that people can start using it more
3. maybe adopt a shorter name, like `dfcli`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist edited a comment on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
Jimexist edited a comment on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939418770
Thanks for raising this issue and question @rupurt.
First off, I don't believe there's any centralized view on what datafusion-cli is or isn't, and there's no process yet to determine that. However I can share some of my thinkings here.
## What datafusion-cli is not
Our ecosystem is full of different tools to manipulate data, each fitting its own niche purpose. In my opinion, datafusion-cli should not try to be yet another general-purpose tool to just manipulate data, especially since datafusion itself is intended to be an embeddable component for other tools (e.g. [cube](https://cube.dev/), [ballista](https://github.com/ballista-compute/ballista), [roapi](https://roapi.github.io/docs/)), to avoid confusion or reduce fragmented tech investment.
Specifically, datafusion-cli is not:
1. a general purpose Python enabled tool to query data, for that you'll have the [Python binding](https://pypi.org/project/datafusion) for datafusion itself or [polar](https://github.com/pola-rs/polars) for its speed and pandas compatibility, both leveraging Arrow and datafusion underneath
2. a command line tool to manipulate small, tabular or structured data, for that you'll have [xsv](https://github.com/BurntSushi/xsv), [jq](https://github.com/stedolan/jq), or [rq](https://github.com/dflemstr/rq), depending on the file formats that one wants
3. a client to an HTTP or GraphQL enabled server backend, for that, you can have [roapi](https://roapi.github.io/docs/) or similar things, or in many cases Spark or Presto is just fine (when data size is large)
Also for 3. please note that (AFAIK) datafusion and datafusion-cli themselves do not concern with distributed computing, i.e. data sharding is something built on top of them - they can only do in-memory, uniformly accessible data manipulation.
## What datafusion-cli can be
Given the above assertions, I believe the place where datafusion-cli can shine is:
1. the data size is large enough so that simple tools like `jq` or `xsv` can't cut it (within reasonable amount of time), but still small enough that can be fit into memory ([EC2 machine has up to 12TB](https://aws.amazon.com/about-aws/whats-new/2021/05/four-ec2-high-memory-instances-with-up-to-12tb-memory-available-with-on-demand-and-savings-plan-purchase-options/)) - if you do care about the speed
2. when there's no need or necessity to keep a long running server or adopt full stack of Spark or Presto cluster due to their high maintenance cost
I think in many cases of ETL you do encounter these type of need for “glue” script that can just run with a simple shell script, either triggered manually, via Airflow, or crontab, etc. And it runs to the end, without any setup needed, takes in one or more parquet files, an SQL script, and spits out target data, either in JSON, CSV, or maybe Parquet.
It can also be a REPL, an interactive shell to quickly verify commands and look into data
## Considerations on API
I think there are two aspects of API to this CLI:
1. the command line API (flags, options, naming, etc.)
2. the query syntax itself
For 2. it's clear that Postgres-compatible SQL is the choice here, but for 1. I don't think we are necessarily bound to the peculiarity of `psql` itself because:
1. it's in many cases specific to Postgres and the fact that it's a client-server architecture
2. `psql` is in many cases used interactively, but when it comes to shell automation, there are all sorts of other scripts in use, e.g. `pg_ctl`, `pg_dumpall`, etc. and here they all map to datafusion-cli
Having said that, I do think commands like `\copy` are already familiarised within users and nice to have, but I admit that in many cases I am still confused on the different flags and behaviors when that command is used interactively versus used as a CLI option. We need to be more consistent here, and just not necessarily consistent with `psql` itself.
## What comes next
Although I'm not recently working on this area, some so call "roadmap" that I have in mind would be:
1. to build a fully abstracted layer of repl parsing so that queries and commands are separated and handled correctly - currently it's kind of a hack
2. to widen the usage by publishing to apt, brew, and possible NuGet registry so that people can start using it more
3. maybe adopt a shorter name, like `dfcli`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
Jimexist commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939418770
Thanks for raising this issue and question @rupurt.
First off, I don't believe there's any centralized view on what datafusion-cli is or isn't, and there's no process yet to determine that. However I can share some of my thinkings here.
## What datafusion-cli is not
Our ecosystem is full of different tools to manipulate data, each fitting its own niche purpose. In my opinion, datafusion-cli should not try to be yet another general-purpose tool to just manipulate data, especially since datafusion itself is intended to be an embeddable component for other tools (e.g. [cube](https://cube.dev/), [ballista](https://github.com/ballista-compute/ballista), [roapi](https://roapi.github.io/docs/)), to avoid confusion or reduce fragmented tech investment.
Specifically, datafusion-cli is not:
1. a general purpose Python enabled tool to query data, for that you'll have the [Python binding](https://pypi.org/project/datafusion) for datafusion itself or [polar](https://github.com/pola-rs/polars) for its speed and pandas compatibility, both leveraging Arrow and datafusion underneath
2. a command line tool to manipulate small, tabular or structured data, for that you'll have [xsv](https://github.com/BurntSushi/xsv), [jq](https://github.com/stedolan/jq), or [rq](https://github.com/dflemstr/rq), depending on the file formats that one wants
3. a client to an HTTP or GraphQL enabled server backend, for that, you can have [roapi](https://roapi.github.io/docs/) or similar things, or in many cases Spark or Presto is just fine (when data size is large)
Also for 3. please note that (AFAIK) datafusion and datafusion-cli themselves do not concern with distributed computing, i.e. data sharding is something built on top of them - they can only do in-memory, uniformly accessible data manipulation.
## What datafusion-cli can be
Given the above assertions, I believe the place where datafusion-cli can shine is:
1. the data size is large enough so that simple tools like `jq` or `xsv` can't cut it (within reasonable amount of time), but still small enough that can be fit into memory ([EC2 machine has up to 12TB](https://aws.amazon.com/about-aws/whats-new/2021/05/four-ec2-high-memory-instances-with-up-to-12tb-memory-available-with-on-demand-and-savings-plan-purchase-options/)) - can you do care about the speed
2. when there's no need or necessity to keep a long running server or adopt full stack of Spark or Presto cluster due to their high maintenance cost
I think in many cases of ETL do you encounter these type of need for “glue” script that can just run with a simple shell script, either triggered manually, via Airflow, or crontab, etc. And it runs to the end, without any setup needed, takes in one or more parquet files, an SQL script, and spits out target data, either in JSON, CSV, or maybe Parquet.
It can also be a REPL, an interactive shell to quickly verify commands and look into data
## Considerations on API
I think there are two aspects of API to this CLI:
1. the command line API (flags, options, naming, etc.)
2. the query syntax itself
For 2. it's clear that Postgres-compatible SQL is the choice here, but for 1. I don't think we are necessarily bound to the peculiarity of `psql` itself because:
1. it's in many cases specific to Postgres and the fact that it's a client-server architecture
2. `psql` is in many cases used interactively, but when it comes to shell automation, there are all sorts of other scripts in use, e.g. `pg_ctl`, `pg_dumpall`, etc. and here they all map to datafusion-cli
Having said that, I do think commands like `\copy` are already familiarised within users and nice to have, but I admit that in many cases I am still confused on the different flags and behaviors when that command is used interactively versus used as a CLI option. We need to be more consistent here, and just not necessarily consistent with `psql` itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
alamb closed issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-941204524
Added my translation in 65910be4e -- additional thoughts welcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-940677866
I think we should PR @Jimexist 's well written comment into #1104 or add to the roadmap.md file after 1104 is merged :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-941204524
Added my translation in 65910be4e -- additional thoughts welcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939917853
FWIW https://github.com/apache/arrow-datafusion/issues/1102 is related to the larger question of "datafusion roadmap"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist edited a comment on issue #1096: Vision for datafusion-cli
Posted by GitBox <gi...@apache.org>.
Jimexist edited a comment on issue #1096:
URL: https://github.com/apache/arrow-datafusion/issues/1096#issuecomment-939418770
Thanks for raising this issue and question @rupurt.
First off, I don't believe there's any centralized view on what datafusion-cli is or isn't, and there's no process yet to determine that. However I can share some of my thinkings here.
## What datafusion-cli is not
Our ecosystem is full of different tools to manipulate data, each fitting its own niche purpose. In my opinion, datafusion-cli should not try to be yet another general-purpose tool to just manipulate data, especially since datafusion itself is intended to be an embeddable component for other tools (e.g. [cube](https://cube.dev/), [ballista](https://github.com/ballista-compute/ballista), [roapi](https://roapi.github.io/docs/)), to avoid confusion or reduce fragmented tech investment.
Specifically, datafusion-cli is not:
1. a general purpose Python enabled tool to query data, for that you'll have the [Python binding](https://pypi.org/project/datafusion) for datafusion itself or [polar](https://github.com/pola-rs/polars) for its speed and pandas compatibility, both leveraging Arrow and datafusion underneath
2. a command line tool to manipulate small, tabular or structured data, for that you'll have [xsv](https://github.com/BurntSushi/xsv), [jq](https://github.com/stedolan/jq), or [rq](https://github.com/dflemstr/rq), depending on the file formats that one wants
3. a client to an HTTP or GraphQL enabled server backend, for that, you can have [roapi](https://roapi.github.io/docs/) or similar things, or in many cases Spark or Presto is just fine (when data size is large)
Also for 3. please note that (AFAIK) datafusion and datafusion-cli themselves do not concern with distributed computing, i.e. data sharding is something built on top of them - they can only do in-memory, uniformly accessible data manipulation.
## What datafusion-cli can be
Given the above assertions, I believe the place where datafusion-cli can shine is:
1. the data size is large enough so that simple tools like `jq` or `xsv` can't cut it (within reasonable amount of time), but still small enough that can be fit into memory ([EC2 machine has up to 12TB](https://aws.amazon.com/about-aws/whats-new/2021/05/four-ec2-high-memory-instances-with-up-to-12tb-memory-available-with-on-demand-and-savings-plan-purchase-options/)) - if you do care about the speed
2. when there's no need or necessity to keep a long running server or adopt full stack of Spark or Presto cluster due to their high maintenance cost
I think in many cases of ETL do you encounter these type of need for “glue” script that can just run with a simple shell script, either triggered manually, via Airflow, or crontab, etc. And it runs to the end, without any setup needed, takes in one or more parquet files, an SQL script, and spits out target data, either in JSON, CSV, or maybe Parquet.
It can also be a REPL, an interactive shell to quickly verify commands and look into data
## Considerations on API
I think there are two aspects of API to this CLI:
1. the command line API (flags, options, naming, etc.)
2. the query syntax itself
For 2. it's clear that Postgres-compatible SQL is the choice here, but for 1. I don't think we are necessarily bound to the peculiarity of `psql` itself because:
1. it's in many cases specific to Postgres and the fact that it's a client-server architecture
2. `psql` is in many cases used interactively, but when it comes to shell automation, there are all sorts of other scripts in use, e.g. `pg_ctl`, `pg_dumpall`, etc. and here they all map to datafusion-cli
Having said that, I do think commands like `\copy` are already familiarised within users and nice to have, but I admit that in many cases I am still confused on the different flags and behaviors when that command is used interactively versus used as a CLI option. We need to be more consistent here, and just not necessarily consistent with `psql` itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org