You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ion-elgreco (via GitHub)" <gi...@apache.org> on 2024/10/13 11:32:50 UTC
[I] Add `pivot` & `unpivot` (melt) to DataFrame [datafusion]
ion-elgreco opened a new issue, #12907:
URL: https://github.com/apache/datafusion/issues/12907
### Is your feature request related to a problem or challenge?
Pivoting and unpivoting is a common use case for data scientists, this is currently missing in the DF api.
### Describe the solution you'd like
Add two methods to DataFrame struct:
- pivot
- possibility to add distinct values in advance (would be optimized into groupby.agg() directly)
- let DataFusion calculate distinct values automatically
- unpivot
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `pivot` & `unpivot` (melt) to DataFrame [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12907:
URL: https://github.com/apache/datafusion/issues/12907#issuecomment-3378471606
@chenkovsky has a PR to implement this:
- https://github.com/apache/datafusion/pull/17946
It looks perhaps similar to the PIVOT from @simonvandel in
- https://github.com/apache/datafusion/pull/17365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `pivot` & `unpivot` (melt) to DataFrame [datafusion]
Posted by "jonmmease (via GitHub)" <gi...@apache.org>.
jonmmease commented on issue #12907:
URL: https://github.com/apache/datafusion/issues/12907#issuecomment-2782922917
Out of curiosity, would it be possible to implement pivot using DataFusion's current extension mechanisms? I had hoped it would be possible as a [UDTF](https://datafusion.apache.org/library-user-guide/adding-udfs.html#adding-a-user-defined-table-function), but it doesn't look like UDTFs can input a table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `pivot` & `unpivot` (melt) to DataFrame [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #12907:
URL: https://github.com/apache/datafusion/issues/12907#issuecomment-2408986299
referenced in https://github.com/apache/datafusion-python/issues/875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `pivot` & `unpivot` (melt) to DataFrame [datafusion]
Posted by "riziles (via GitHub)" <gi...@apache.org>.
riziles commented on issue #12907:
URL: https://github.com/apache/datafusion/issues/12907#issuecomment-2784426225
@jonmmease , not sure if it's helpful, but Ibis has a pure Python implementation that works on DataFusion dataframes:
https://github.com/ibis-project/ibis/blob/20bec137463d0dba71e741247624119b7a7b3452/ibis/expr/types/relations.py#L4005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `pivot` & `unpivot` (melt) to DataFrame [datafusion]
Posted by "simonvandel (via GitHub)" <gi...@apache.org>.
simonvandel commented on issue #12907:
URL: https://github.com/apache/datafusion/issues/12907#issuecomment-3378733171
@chenkovsky your PR seems better than my attempt, so let's go for that. I'll be happy to use the LogicalPlanBuilder::pivot from your PR in my pipe operator PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `pivot` & `unpivot` (melt) to DataFrame [datafusion]
Posted by "osipovartem (via GitHub)" <gi...@apache.org>.
osipovartem commented on issue #12907:
URL: https://github.com/apache/datafusion/issues/12907#issuecomment-5558807571
I would like to pick up the UNPIVOT half as a focused first PR. I maintain a downstream implementation, but rather than revive the previous 1,000+ line combined change, I plan to split it into:
1. a reusable `LogicalPlanBuilder::unpivot` transformation,
2. a thin DataFrame API wrapper, and
3. SQL `TableFactor::Unpivot` lowering with focused tests.
The implementation will use ordinary projections + `UNION ALL`, avoid dependencies on scalar UDFs, cover INCLUDE/EXCLUDE NULLS, aliases, type coercion, and validation, and leave PIVOT for a separate follow-up. I will preserve attribution to the prior work where code is reused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org