You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/07 12:04:59 UTC
[GitHub] [arrow-datafusion] kmitchener commented on issue #3058: EPIC: Improve Documentation, Tutorials, and Examples
kmitchener commented on issue #3058:
URL: https://github.com/apache/arrow-datafusion/issues/3058#issuecomment-1207393276
As a relatively new user of DataFusion, I agree the docs on the site are pretty bad, but not so bad that I wasn't able to understand it without being overwhelmed :) However, there's plenty of room for improvement and I've been doing a few things related to this:
* reviewing all the documentation-tagged issues in this project
* reviewing other similar project's documentation to see how it's structured, what's helpful to me as a newbie to their project, what's confusing about their documentation
* trying (locally) different structure for documentation based on the above
I've come up with a few changes that I think will help:
* realize that the DataFusion library itself is the product here, not DataFusion-CLI. DataFusion-CLI is a tool for demonstrating the power of the library quickly and easily for potential users (among other uses, but to me this seems to be the primary use case -- if it's not, it's worth clarifying what the intended use of DataFusion-CLI is).
* as such, DataFusion-CLI should have a supporting role in the documentation
* as such, there should be more examples how potential new users can use DataFusion-CLI to quickly demonstrate for themselves how DataFusion library can help them
* therefore there should be multiple examples how to run DataFusion-CLI, how to register a variety of data into DataFusion context, and the power of the SQL -- examples should cover loading data from local, from object store like S3, partitioned and not, different formats, etc. Queries should be run. Explain plans and `explain analyze` should be shown.
* probably CLI itself will need some changes to make it easier to use it against S3 or Azure Blob Storage. It should be able to parse the location given in a `create external table` command to automatically register an object store using sensible default authentication methods. for example, if I already have my environment configured to use the AWS CLI, then I should be able to startup DataFusion-CLI and run `create external table test stored as parquet location 's3::/my-bucket/content'` and it should "just work".
* there should be a separate User Guide and a Developer Guide (or maybe call it a "Contributor Guide"?)
* yes, in the User Guide, I think the functions we have in DataFusion should be documented. for new prospective users browsing docs, it's important to see the wide variety of useful functions that exist.
* related to this, some of the documentation makes it appear that DataFusion is _less_ capable than it actually is -- in particular, the bit that describes the very basic SQL syntax that works (implying that more complex SQL won't work)
* ETL is called out as particular use case for DataFusion, but none of the examples demonstrate ETL pipelines. In my opinion, definitely need some examples or even just write-ups about how DataFusion can work as an ETL tool.
It's still very much a WIP, but I have a structure that I think mostly makes sense in this branch: https://github.com/kmitchener/arrow-datafusion/tree/doc-improvements (related to PR #3005 )
The above incorporates some of the thoughts from #1821 and #1814 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org