You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/31 11:09:48 UTC

[GitHub] [arrow] neopaf opened a new issue, #15138: Clustered By -- how?

neopaf opened a new issue, #15138:
URL: https://github.com/apache/arrow/issues/15138

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   As I understand 
   https://cloudxlab.com/blog/bucketing-clustered-by-and-cluster-by/
   
   Hive supports "CLUSTERED BY (cols)" approach, where data are split into `0000<bucket>` files.
   Also Hive supports "PARTITIONED BY (cols)".
   
   In Arrow we have `arrow::dataset` that can be configured to write to different partitions like [this](https://arrow.apache.org/docs/cpp/dataset.html).
   
   I can't seem to find a way to write data, splitting them into relevant bucket-files.
   
   I guess, I need to define additional [custom] partitioning.
   
   But can't seem to find relevant hash function.
   
   Maybe somebody already walked that path and can share some insights (if not code)?
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorisvandenbossche commented on issue #15138: Clustered By -- how?

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on issue #15138:
URL: https://github.com/apache/arrow/issues/15138#issuecomment-1387388664

   Also Apache Iceberg has a "bucket[N]" transform for partitioning: https://iceberg.apache.org/spec/#partitioning


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] westonpace commented on issue #15138: [C++] Clustered By -- how?

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #15138:
URL: https://github.com/apache/arrow/issues/15138#issuecomment-1399076522

   Internally we do this by hashing the column.  There is a PR under way (though it could use some review) to add a new hash compute function (https://github.com/apache/arrow/pull/13487).  If that were to merge then you could approximate this pretty well by partitioning on bit_wise_and(hash(x), mask) where `mask` is something like `0x7` (for 8 groups) or `0xF` (for 16 groups), etc.  To support non-powers-of-two groups we would need modulo.  There is a PR in place for modulo but it seems to have stalled.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] neopaf commented on issue #15138: [C++] Clustered By -- how?

Posted by "neopaf (via GitHub)" <gi...@apache.org>.
neopaf commented on issue #15138:
URL: https://github.com/apache/arrow/issues/15138#issuecomment-1400101599

   I guess, for my purpose the ideal way to inject is this: https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest+V2
   
   Hence I guess I'll need to switch to java world :(
   
   thanks, @westonpace !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] neopaf closed issue #15138: [C++] Clustered By -- how?

Posted by "neopaf (via GitHub)" <gi...@apache.org>.
neopaf closed issue #15138: [C++] Clustered By -- how?
URL: https://github.com/apache/arrow/issues/15138


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org