You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/07/26 14:52:23 UTC

[GitHub] [arrow-datafusion-python] alamb opened a new issue, #440: [DISCUSSION] We need a Hero for datafusion-python

alamb opened a new issue, #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440

   ## What this project could be
   
   I think this project needs someone who wants to make a world class python dataframe library and user experience take the helm.  I will argue why I think this is a compelling opportunity to make a great piece of technology and have a wide impact across the data analytic space:
   
   ## What this project could be
   
   I think this project could be one of the most widely used data analysis libraries out there. Imagine a system that allows **BOTH** a fast dataframe API (ala pol.rs) but also first class  SQL support (ala duckdb) that are both screaming fast (due to all the effort that goes into https://github.com/apache/arrow-datafusion) as well as easy to plug into the eco system (arrow / parquet) and extensible (UDFS, UDAs, etc)
   
   [DataFusion](https://arrow.apache.org/datafusion/) already posts great benchmark numbers, and I will post datafusion 28.0.0 benchmark when we have them.
   
   ## How is this different than the mission of DataFusion?
   [DataFusion](https://arrow.apache.org/datafusion/) is a great project but is currently focused on building the core analytic engine:
   
   > DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in [Rust](http://rustlang.org/), using the [Apache Arrow](https://arrow.apache.org/) in-memory format.
   
   ![image](https://github.com/apache/arrow-datafusion-python/assets/490673/16941b00-830a-4488-adf5-13a7e86be977)
   
   This repository contains basic python bindings, but the user experience (UX) could be improved in so many ways. 
   
   
   ## The opportunity
   
   This would be a great opportunity for someone to:
   
   1. Build some really cool technology
   1. Learn how to help grow an open source project and community with help and guidance from the rest of the DataFusion community
   2. Learn about analytic database technology, Arrow, etc
   3. Influence the direction of Development in DataFusion
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [arrow-datafusion-python]

Posted by "dlr2 (via GitHub)" <gi...@apache.org>.
dlr2 commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1919480158

   I am also late to this, but as I am trying to evaluate datafusion (Python) I can give some of my input. I am sure that folks know that the documentation is scant, with most API functions having no more than the method name and args (auto extracted from sphinx). 
   
   My next idea was to test the SQL vs native expression filtering. Got the SQL to work, but I cannot see how to use an 'and' expr/function. As this is a reserved word I saw no way to apply it or import it. Would the native expr filtering be faster than the equivalent SQL?
   
   So yes, complete examples (showing all the imports, etc) for all the functions and expressions would be great. Hope this is useful


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] alamb commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1655724191

   I propose we leave the the decision of where to take this project and what to focus on to whatever hero(s) step forward. What I think `datafusion-python` needs is someone to invest the time to drive it forward, and the path to take, as in all open source projects, would be largely influenced by the contributors. 
   
   
   > I'd like to propose that [ibis](https://ibis-project.org/) is the library that solves the problem of bringing a delightful and expressive DataFrame API to [DataFusion](https://ibis-project.org/backends/datafusion/) with no loss of SQL functionality. You can actually mix and match [ibis expressions and SQL](https://ibis-project.org/reference/expressions/tables/#ibis.expr.types.relations.Table.sql) to your heart's content.
   
   Thank you @cpcloud  -- this is an excellent idea and it would be awesome to see the DataFusion ibis backend become more full featured. 
   
   I think a very reasonable alternative reality is that "datafusion-python" remains a thin binding on top of datafusion, and the delightful user experience comes via ibis.
   
   > What do say we ... COALESCE πŸ˜‰ around ibis as the DataFrame API for DataFusion?
   
   That is one of cleverest  summaries I have seen in a long time. Nicely done πŸ‘ 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] mesejo commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "mesejo (via GitHub)" <gi...@apache.org>.
mesejo commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1708045520

   Folks! I've created some issues to tackle the missing functions in the Python bindings. 
   
   - https://github.com/apache/arrow-datafusion-python/issues/463
   - https://github.com/apache/arrow-datafusion-python/issues/466
   
   These are a perfect fit for a [good first issue](https://label.dev/good-first-issue/), so contributions are more than welcome. (@alamb perhaps we could label the issues as such and promote them on Twitter to increase the involvement with the project?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] magarick commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "magarick (via GitHub)" <gi...@apache.org>.
magarick commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1687280393

   Hi everyone. I'm happy to help out with this. I think it might be a good idea to get a sense for what people think this should ultimately look like as well as what features they think a good DataFrame library should have. To that end, I've started this issue which hopefully will help gather ideas and fodder for documentation https://github.com/apache/arrow-datafusion-python/issues/462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] francis-du commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "francis-du (via GitHub)" <gi...@apache.org>.
francis-du commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1655545243

   I will continue the unfinished work when I have time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [arrow-datafusion-python]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1879239594

   > Is there anyone currently adding pyi files for datafusion-python? I have experience in this area and I would like to be involved in this work
   
   Thanks @woxiaosa  -- I do not know of anyone currently adding pyi files


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #440:
URL: https://github.com/apache/datafusion-python/issues/440#issuecomment-2473637217

   Also a big thank you to @mesejo !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

[GitHub] [arrow-datafusion-python] alamb commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1667890731

   > For solving 1. we could follow the PyO3 [guide](https://pyo3.rs/v0.19.1/python_typing_hints) and add information in .pyi files.
   
   That seems like a very good idea to me
   
   > For 2. one alternative is to wrap the methods and functions in Python and add docstrings to them (similar to what polars do, see [this](https://github.com/pola-rs/polars/blob/main/py-polars/polars/expr/expr.py#L6980-L7007) for example).
   
   I think this is likewise a great idea
   
   Thank you @mesejo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "lostmygithubaccount (via GitHub)" <gi...@apache.org>.
lostmygithubaccount commented on issue #440:
URL: https://github.com/apache/datafusion-python/issues/440#issuecomment-2108024950

   > This PR does not close https://github.com/apache/datafusion-python/issues/440 but it helps to address one part of it.
   
   somebody at GitHub is going to use this as evidence for LLM-based issue closing instead of the current rules


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

[GitHub] [arrow-datafusion-python] cpcloud commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "cpcloud (via GitHub)" <gi...@apache.org>.
cpcloud commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1653362679

   This looks like a great idea!
   
   I'd like to propose that [ibis](https://ibis-project.org) is the library that solves the problem of bringing a delightful and expressive DataFrame API to [DataFusion](https://ibis-project.org/backends/datafusion/) with no loss of SQL functionality. You can actually mix and match ibis expressions and SQL to your heart's content.
   
   As @mesejo mentioned, they've been making [great contributions to the DataFusion backend](https://github.com/ibis-project/ibis/pulls?q=is%3Apr+sort%3Aupdated-desc+datafusion+author%3Amesejo+is%3Aclosed) so there's currently some momentum that we can take advantage of.
   
   I'll be up front about it: there's still a lot of work to do, the DataFusion backend is missing a lot of functionality.
   
   The good news is that we've made really easy to see what functionality is missing from any given backend using [our backend support matrix app](https://ibis-project.org/backends/_support_matrix/).
   
   Anyone can take a pass at implementing the operations that have a 🚫 in the `datafusion` column. Some operations will be more challenging than others, and the ibis maintainers (@kszucs, @gforsyth, @jcrist and myself) are here to help.
   
   What do say we ... `COALESCE` :wink: around ibis as *the* DataFrame API for DataFusion?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] alamb commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1710488395

   Thanks @mesejo  -- I marked the tickets as good first issue and posted a tweet: https://twitter.com/andrewlamb1111/status/1699827809462440353


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #440:
URL: https://github.com/apache/datafusion-python/issues/440#issuecomment-2472427847

   Given the work of @timsaucer @Michael-J-Ward @kosiew and @ion-elgreco  have done with this repository recently, I think we could say "the call has been answered here" 🦸  πŸš€ 
   
   I think we should close it 
   
   Thank you again for all your hard work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [arrow-datafusion-python]

Posted by "woxiaosa (via GitHub)" <gi...@apache.org>.
woxiaosa commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1876233169

   > > For solving 1. we could follow the PyO3 [guide](https://pyo3.rs/v0.19.1/python_typing_hints) and add information in .pyi files.
   > 
   > That seems like a very good idea to me
   > 
   > > For 2. one alternative is to wrap the methods and functions in Python and add docstrings to them (similar to what polars do, see [this](https://github.com/pola-rs/polars/blob/main/py-polars/polars/expr/expr.py#L6980-L7007) for example).
   > 
   > I think this is likewise a great idea
   > 
   > Thank you @mesejo
   
   Is there anyone currently adding pyi files for datafusion-python? I have experience in this area and I would like to be involved in this work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] mesejo commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "mesejo (via GitHub)" <gi...@apache.org>.
mesejo commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1666842213

   @alamb Thanks for the feedback 
   
   > Maybe you have time to pretend you are a first time user and and figure out what is not clear or where the rough edges are. Ideally you could turn that experience into a guide to help others.
   
   I open a [PR](https://github.com/apache/arrow-datafusion-python/pull/445) with a draft for the User Guide πŸ˜ƒ. While I was writing the guide, I noticed two issues that have a huge impact on the UX and are simple to solve:
   
   1. The IDE cannot provide hints (or autocompletion) because there is no typing information.  
   2. There are no examples of how to use each method (or function)
   
   For solving 1. we could follow the PyO3 [guide](https://pyo3.rs/v0.19.1/python_typing_hints) and add information in .pyi files. 
   
   For 2. one alternative is to wrap the methods and functions in Python and add docstrings to them (similar to what polars do, see [this](https://github.com/pola-rs/polars/blob/main/py-polars/polars/expr/expr.py#L6980-L7007) for example). 
   
   What are your thoughts?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] mesejo commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "mesejo (via GitHub)" <gi...@apache.org>.
mesejo commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1652213194

   I'm willing to lend a hand πŸ˜„. Are there any requirements πŸ˜… ? Recently I've been expanding the coverage of DataFusion by [ibis](https://github.com/ibis-project/ibis) backend for DataFusion. In the past, I've had (minor) contributions to projects such as dask, xarray, geopandas, and eland. Let me know how I can help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #440:
URL: https://github.com/apache/datafusion-python/issues/440#issuecomment-2479173078

   Are we okay to close this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #440: [DISCUSSION] We need a Hero for datafusion-python
URL: https://github.com/apache/datafusion-python/issues/440


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

[GitHub] [arrow-datafusion-python] devinjdangelo commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1684140919

   I’m late this discussion (and new to this project in general), but the contributions I’ve been focused on over the past month or so have been aimed at solving some of the gaps I see as a heavy python user with a Data Science / Engineering background. Particularly for ETL usecases it needs to be easy to move and transform data between various formats and Object stores leveraging every core available to the maximum extent possible. Default options need to be well tuned since most of these users imo won’t give DataFusion a second look if they run their job and it is much slower than polars or XYZ tool they use currently.
   
   I haven’t gotten to actually looking much at the python interface yet, but it is on my list. 
   
   I am very much on board with the vision you describe @alamb. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #440:
URL: https://github.com/apache/datafusion-python/issues/440#issuecomment-2107989366

   I think github got a little excited about closing this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #440: [DISCUSSION] We need a Hero for datafusion-python
URL: https://github.com/apache/datafusion-python/issues/440


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

[GitHub] [arrow-datafusion-python] lostmygithubaccount commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "lostmygithubaccount (via GitHub)" <gi...@apache.org>.
lostmygithubaccount commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1656286966

   just to throw out an idea related to this:
   
   > I think a very reasonable alternative reality is that "datafusion-python" remains a thin binding on top of datafusion, and the delightful user experience comes via ibis.
   
   if we agree Ibis is a delightful dataframe API and we can close the gaps in the DataFusion backend, then you could avoid a lot of work in defining a new dataframe API by wrapping Ibis so that code looks like:
   
   ```python
   [ins] In [3]: t = datafusion.read_parquet("penguins.parquet")
   
   [ins] In [4]: t
   Out[4]:
   DatabaseTable: _ibis_read_parquet_pnfkuttmizcmjk7trfkv5bhfse
     species           string
     island            string
     bill_length_mm    float64
     bill_depth_mm     float64
     flipper_length_mm int64
     body_mass_g       int64
     sex               string
     year              int64
   
   [ins] In [5]: datafusion.options.interactive = True
   
   [ins] In [6]: t
   Out[6]:
   ┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
   ┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
   ┑━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
   β”‚ string  β”‚ string    β”‚ float64        β”‚ float64       β”‚ int64             β”‚ int64       β”‚ string β”‚ int64 β”‚
   β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€
   β”‚ Adelie  β”‚ Torgersen β”‚           39.1 β”‚          18.7 β”‚               181 β”‚        3750 β”‚ male   β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           39.5 β”‚          17.4 β”‚               186 β”‚        3800 β”‚ female β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           40.3 β”‚          18.0 β”‚               195 β”‚        3250 β”‚ female β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚            nan β”‚           nan β”‚              NULL β”‚        NULL β”‚ NULL   β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           36.7 β”‚          19.3 β”‚               193 β”‚        3450 β”‚ female β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           39.3 β”‚          20.6 β”‚               190 β”‚        3650 β”‚ male   β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           38.9 β”‚          17.8 β”‚               181 β”‚        3625 β”‚ female β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           39.2 β”‚          19.6 β”‚               195 β”‚        4675 β”‚ male   β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           34.1 β”‚          18.1 β”‚               193 β”‚        3475 β”‚ NULL   β”‚  2007 β”‚
   β”‚ Adelie  β”‚ Torgersen β”‚           42.0 β”‚          20.2 β”‚               190 β”‚        4250 β”‚ NULL   β”‚  2007 β”‚
   β”‚ …       β”‚ …         β”‚              … β”‚             … β”‚                 … β”‚           … β”‚ …      β”‚     … β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”˜
   
   [ins] In [7]: t.group_by(["species", "island"]).agg(datafusion._.count())
   Out[7]:
   ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
   ┃ species   ┃ island    ┃ CountStar(_ibis_read_parquet_pnfkuttmizcmjk7trfkv5bhfse) ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
   β”‚ string    β”‚ string    β”‚ int64                                                    β”‚
   β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
   β”‚ Adelie    β”‚ Biscoe    β”‚                                                       44 β”‚
   β”‚ Adelie    β”‚ Torgersen β”‚                                                       52 β”‚
   β”‚ Adelie    β”‚ Dream     β”‚                                                       56 β”‚
   β”‚ Chinstrap β”‚ Dream     β”‚                                                       68 β”‚
   β”‚ Gentoo    β”‚ Biscoe    β”‚                                                      124 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   
   [ins] In [8]: t.group_by(["species", "island"]).agg(datafusion._.count().name("count"))
   Out[8]:
   ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┓
   ┃ species   ┃ island    ┃ count ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━┩
   β”‚ string    β”‚ string    β”‚ int64 β”‚
   β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€
   β”‚ Adelie    β”‚ Biscoe    β”‚    44 β”‚
   β”‚ Adelie    β”‚ Torgersen β”‚    52 β”‚
   β”‚ Chinstrap β”‚ Dream     β”‚    68 β”‚
   β”‚ Gentoo    β”‚ Biscoe    β”‚   124 β”‚
   β”‚ Adelie    β”‚ Dream     β”‚    56 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”˜
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] alamb commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1655734173

   > I'm willing to lend a hand πŸ˜„. Are there any requirements πŸ˜… ? Recently I've been expanding the coverage of DataFusion by [ibis](https://github.com/ibis-project/ibis). In the past, I've had (minor) contributions to projects such as dask, xarray, geopandas, and eland. Let me know how I can help.
   
   Thank you @mesejo  -- that is great. Like many projects, I think what would be most valuable in this project is 
   1. Reviewing PRs and encouraging more involvment
   2. Ensuring the project is easy to both use and contribute such as #432 
   
   Maybe you have time to pretend you are a first time user and and figure out what is not clear or where the rough edges are. Ideally you could turn that experience into a guide to help others.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion-python] kylebarron commented on issue #440: [DISCUSSION] We need a Hero for datafusion-python

Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on issue #440:
URL: https://github.com/apache/arrow-datafusion-python/issues/440#issuecomment-1668113514

   Just a note that with manual `.pyi` files you have the endless problem of ensuring that the `.pyi` files and code match up correctly. Wrapping every rust function in a pure-python function works as polars does but also incurs a ton of overhead. The long-term solution is if pyo3 can emit python type files automatically, as wasm-bindgen does for TypeScript, but that's likely far off


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSSION] We need a Hero for datafusion-python [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #440:
URL: https://github.com/apache/datafusion-python/issues/440#issuecomment-2479174644

   Closing per recommendation from @alamb and no additional requests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org