You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/12/15 23:41:00 UTC

[GitHub] [arrow] wjones127 opened a new pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

wjones127 opened a new pull request #11970:
URL: https://github.com/apache/arrow/pull/11970


   Created an initial draft here and will place similar guidance in PyArrow and R arrow documentation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] wjones127 commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
wjones127 commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770946135



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,46 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets has two aspects that affect performance: it increases the number of
+files and it creates a directory structure around the files. Both of these have benefits
+as well as costs. Depending on the configuration and the size of your dataset, the costs 
+can outweigh the benefits. 
+
+Because partitions split up the dataset into multiple files, partitioned datasets can be 
+read and written with parallelism. However, each additional file adds a little overhead in 
+processing for filesystem interaction. It also increases the overall dataset size since 
+each file has some shared metadata. For example, each parquet file contains the schema and
+group-level statistics. The number of partitions is a floor for the number of files. If 
+you partition a dataset by date with a year of data, you will have at least 365 files. If 
+you further partition by another dimension with 1,000 unique values, you will have up to 
+365,000 files. This fine of partitioning often leads to small files that mostly consist of
+metadata.
+
+Partitioned datasets create nested folder structures, and those allow us to prune which 
+files are loaded in a scan. However, this adds overhead to discovering files in the dataset,
+as we'll need to recursively "list directory" to find the data files. These operations may 
+be particularly expensive if you are using an object store filesystem such as S3. Too fine
+partitions can cause problems here: Partitioning a dataset by date for a years worth
+of data will require 365 list calls to find all the files; adding another column with 
+cardinality 1,000 will make that 365,365 calls.

Review comment:
       Yeah I'm probably wrong at that then. I've heard this idea in the Spark context, but it might not be recursive part as much as pagination through a web API that makes this slow. I will remove this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996988701


   Benchmark runs are scheduled for baseline = 670af338bc740888bffea65b28ee2bcc065b555a and contender = 7cf7442633bc887c62370f810f9bce976e6f083e. 7cf7442633bc887c62370f810f9bce976e6f083e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34849734117a46cd835996df060045da...5480a6881b164fc2b31f96847cd60007/)
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7f7ef9291e164956b625905c56480709...02d5aa20b01f4f7a8d8e513df5576dc1/)
   [Finished :arrow_down:0.22% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/17abfcd29dc04eb3b14b1c3f22c4f509...741422a2b0004a419a20c009a7393e1b/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996988701


   Benchmark runs are scheduled for baseline = 670af338bc740888bffea65b28ee2bcc065b555a and contender = 7cf7442633bc887c62370f810f9bce976e6f083e. 7cf7442633bc887c62370f810f9bce976e6f083e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34849734117a46cd835996df060045da...5480a6881b164fc2b31f96847cd60007/)
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7f7ef9291e164956b625905c56480709...02d5aa20b01f4f7a8d8e513df5576dc1/)
   [Finished :arrow_down:0.22% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/17abfcd29dc04eb3b14b1c3f22c4f509...741422a2b0004a419a20c009a7393e1b/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-995301901


   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW
   
   Opening JIRAs ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename pull request title in the following format?
   
       ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770223498



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,25 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets can improve performance when reading datasets, but have several 
+potential costs when reading and writing:
+
+#. Can significantly increase the number of files to write. The number of partitions is a
+   floor for the number of files in a dataset. If you partition a dataset by date with a 
+   year of data, you will have at least 365 files. If you further partition by another
+   dimension with 1,000 unique values, you will have 365,000 files. This can make it slower
+   to write and increase the size of the overall dataset because each file has some fixed
+   overhead. For example, each file in parquet dataset contains the schema.
+#. Multiple partitioning columns can produce deeply nested folder structures which are slow
+   to navigate because they require many recusive "list directory" calls to discover files.
+   These operations may be particularly expensive if you are using an object store 
+   filesystem such as S3. One workaround is to combine multiple columns into one for
+   partitioning. For example, instead of a schema like /year/month/day/ use /YYYY-MM-DD/.
+ 
+

Review comment:
       I suppose compression complicates things too :shrug: 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ldacey commented on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ldacey commented on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-1032885971


   Any recommendations for a couple of scenarios that I run into often:
   
   1) Data gets downloaded often (hourly, every 30 minutes) and saved to a dataset. My current approach is to get the list of fragments which were written, then figure out the partitions (`ds._get_partition_keys(frag.partition_expression)`), read all of the data in those partitions, and then resave the dataset using "delete_matching". I suppose this question would be about data consolidation / small file cleanup best practices in general.
   
   2) Any thoughts on partitions based on ETL schedules? I have [used an old Airflow article](https://maximebeauchemin.medium.com/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a) as reference when considering how I partition data. This normally creates small files for a lot of sources, but one benefit is that I can always clear an Airflow task and it will overwrite any existing data which existed for that schedule run.
   
   <img width="725" alt="image" src="https://user-images.githubusercontent.com/12517855/153043954-64428ee1-80b8-48f6-ac73-23b41643bd70.png">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770165344



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,25 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets can improve performance when reading datasets, but have several 
+potential costs when reading and writing:
+
+#. Can significantly increase the number of files to write. The number of partitions is a
+   floor for the number of files in a dataset. If you partition a dataset by date with a 
+   year of data, you will have at least 365 files. If you further partition by another
+   dimension with 1,000 unique values, you will have 365,000 files. This can make it slower
+   to write and increase the size of the overall dataset because each file has some fixed
+   overhead. For example, each file in parquet dataset contains the schema.
+#. Multiple partitioning columns can produce deeply nested folder structures which are slow
+   to navigate because they require many recusive "list directory" calls to discover files.
+   These operations may be particularly expensive if you are using an object store 
+   filesystem such as S3. One workaround is to combine multiple columns into one for
+   partitioning. For example, instead of a schema like /year/month/day/ use /YYYY-MM-DD/.
+ 
+

Review comment:
       Both of these are in the "cons" section.  It might be worth adding a bit more body to "can improve the performance when reading datasets".
   
   There are two advantages (but really only one):
   
    * We need multiple files to read in parallel.
    * Smaller partitions allow for more selective queries.  E.g. we can load less data from the disk.
   
   We should also mention (here or elsewhere) that everything that applies here for # of files also applies for # of record batchs (or # of row groups in parquet).  It's possible to have 1 file with way too many row groups and get similar performance issues.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] wjones127 commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
wjones127 commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770922505



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,25 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets can improve performance when reading datasets, but have several 
+potential costs when reading and writing:
+
+#. Can significantly increase the number of files to write. The number of partitions is a
+   floor for the number of files in a dataset. If you partition a dataset by date with a 
+   year of data, you will have at least 365 files. If you further partition by another
+   dimension with 1,000 unique values, you will have 365,000 files. This can make it slower
+   to write and increase the size of the overall dataset because each file has some fixed
+   overhead. For example, each file in parquet dataset contains the schema.
+#. Multiple partitioning columns can produce deeply nested folder structures which are slow
+   to navigate because they require many recusive "list directory" calls to discover files.
+   These operations may be particularly expensive if you are using an object store 
+   filesystem such as S3. One workaround is to combine multiple columns into one for
+   partitioning. For example, instead of a schema like /year/month/day/ use /YYYY-MM-DD/.
+ 
+

Review comment:
       @westonpace rewrote based on your feedback.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996982367






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ateucher commented on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ateucher commented on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-997064972


   Thanks for this! I opened https://issues.apache.org/jira/browse/ARROW-15069 struggling with this issue. The response could have been (and would have been in other projects) "you're doing it wrong, not an issue", but instead you took it seriously and expanded the documentation. Thanks for your commitment to your community.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on pull request #11970: MINIOR: [Docs] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996981145


   I'm going to make a JIRA real quick since this is technically too large for the rules on minor PRs now that we are putting it in three places: https://github.com/apache/arrow/blob/5cabd31c90dbb32d87074928f68bf5d6e97e37c6/CONTRIBUTING.md#minor-fixes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996988701


   Benchmark runs are scheduled for baseline = 670af338bc740888bffea65b28ee2bcc065b555a and contender = 7cf7442633bc887c62370f810f9bce976e6f083e. 7cf7442633bc887c62370f810f9bce976e6f083e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34849734117a46cd835996df060045da...5480a6881b164fc2b31f96847cd60007/)
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7f7ef9291e164956b625905c56480709...02d5aa20b01f4f7a8d8e513df5576dc1/)
   [Finished :arrow_down:0.22% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/17abfcd29dc04eb3b14b1c3f22c4f509...741422a2b0004a419a20c009a7393e1b/)
   Buildkite builds:
   [Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/122| `7cf74426` ec2-t3-xlarge-us-east-2>
   [Failed] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/110| `7cf74426` ursa-i9-9960x>
   [Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/113| `7cf74426` ursa-thinkcentre-m75q>
   [Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/121| `670af338` ec2-t3-xlarge-us-east-2>
   [Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/109| `670af338` ursa-i9-9960x>
   [Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/112| `670af338` ursa-thinkcentre-m75q>
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] wjones127 commented on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
wjones127 commented on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-1032894850


   @ldacey Thanks for the question. Would you mind moving it to either: (1) our GitHub issues section, (2) our user mailing list (user@arrow.apache.org), or (3) our [Jira](https://issues.apache.org/jira/projects/ARROW/issues/)? We like to keep usage discussion in one of those three places so it's easier for others to find.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace closed pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
westonpace closed pull request #11970:
URL: https://github.com/apache/arrow/pull/11970


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot commented on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ursabot commented on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996988701


   Benchmark runs are scheduled for baseline = 670af338bc740888bffea65b28ee2bcc065b555a and contender = 7cf7442633bc887c62370f810f9bce976e6f083e. 7cf7442633bc887c62370f810f9bce976e6f083e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34849734117a46cd835996df060045da...5480a6881b164fc2b31f96847cd60007/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7f7ef9291e164956b625905c56480709...02d5aa20b01f4f7a8d8e513df5576dc1/)
   [Scheduled] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/17abfcd29dc04eb3b14b1c3f22c4f509...741422a2b0004a419a20c009a7393e1b/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770166418



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,25 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets can improve performance when reading datasets, but have several 
+potential costs when reading and writing:
+
+#. Can significantly increase the number of files to write. The number of partitions is a
+   floor for the number of files in a dataset. If you partition a dataset by date with a 
+   year of data, you will have at least 365 files. If you further partition by another
+   dimension with 1,000 unique values, you will have 365,000 files. This can make it slower
+   to write and increase the size of the overall dataset because each file has some fixed
+   overhead. For example, each file in parquet dataset contains the schema.
+#. Multiple partitioning columns can produce deeply nested folder structures which are slow
+   to navigate because they require many recusive "list directory" calls to discover files.
+   These operations may be particularly expensive if you are using an object store 
+   filesystem such as S3. One workaround is to combine multiple columns into one for
+   partitioning. For example, instead of a schema like /year/month/day/ use /YYYY-MM-DD/.
+ 
+

Review comment:
       Probably a reasonable "rule of thumb" might be to structure files so that each column of data is at least 4MB large.  This is somewhat arbitrary when it comes to data/metadata ration but 4MB is also around the point where an HDD's sequential vs random reads tradeoff starts to fall off.  Although for bitmaps the requirement for 32 million rows can be a bit extreme / difficult to satisfy.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770936440



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,46 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets has two aspects that affect performance: it increases the number of
+files and it creates a directory structure around the files. Both of these have benefits
+as well as costs. Depending on the configuration and the size of your dataset, the costs 
+can outweigh the benefits. 
+
+Because partitions split up the dataset into multiple files, partitioned datasets can be 
+read and written with parallelism. However, each additional file adds a little overhead in 
+processing for filesystem interaction. It also increases the overall dataset size since 
+each file has some shared metadata. For example, each parquet file contains the schema and
+group-level statistics. The number of partitions is a floor for the number of files. If 
+you partition a dataset by date with a year of data, you will have at least 365 files. If 
+you further partition by another dimension with 1,000 unique values, you will have up to 
+365,000 files. This fine of partitioning often leads to small files that mostly consist of
+metadata.
+
+Partitioned datasets create nested folder structures, and those allow us to prune which 
+files are loaded in a scan. However, this adds overhead to discovering files in the dataset,
+as we'll need to recursively "list directory" to find the data files. These operations may 
+be particularly expensive if you are using an object store filesystem such as S3. Too fine
+partitions can cause problems here: Partitioning a dataset by date for a years worth
+of data will require 365 list calls to find all the files; adding another column with 
+cardinality 1,000 will make that 365,365 calls.
+
+The most optimal partitioning layout will depend on your data, access patterns, and which
+systems will be reading the data. Most systems, including Arrow, should work across a 
+range of file sizes and partitioning layouts, but there are extremes you should avoid. To 
+avoid pathological behavior, keep to these guidelines:

Review comment:
       I like this idea but the wording seems like you will always avoid bad behavior if you follow these rules.  It's entirely possible to design a 50MB file with poor row groupings (although you do discuss that in the next paragraph).  Maybe we could word it as "These guidelines can help avoid some known worst case situations" or something like that.

##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,46 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets has two aspects that affect performance: it increases the number of
+files and it creates a directory structure around the files. Both of these have benefits
+as well as costs. Depending on the configuration and the size of your dataset, the costs 
+can outweigh the benefits. 
+
+Because partitions split up the dataset into multiple files, partitioned datasets can be 
+read and written with parallelism. However, each additional file adds a little overhead in 
+processing for filesystem interaction. It also increases the overall dataset size since 
+each file has some shared metadata. For example, each parquet file contains the schema and
+group-level statistics. The number of partitions is a floor for the number of files. If 
+you partition a dataset by date with a year of data, you will have at least 365 files. If 
+you further partition by another dimension with 1,000 unique values, you will have up to 
+365,000 files. This fine of partitioning often leads to small files that mostly consist of
+metadata.
+
+Partitioned datasets create nested folder structures, and those allow us to prune which 
+files are loaded in a scan. However, this adds overhead to discovering files in the dataset,
+as we'll need to recursively "list directory" to find the data files. These operations may 
+be particularly expensive if you are using an object store filesystem such as S3. Too fine
+partitions can cause problems here: Partitioning a dataset by date for a years worth
+of data will require 365 list calls to find all the files; adding another column with 
+cardinality 1,000 will make that 365,365 calls.

Review comment:
       Ironically I think this is actually worse on local filesystems than it is on S3.  S3 supports a recursive query (and we use it I'm pretty sure) so we only actually do a single list directory call.  Maybe just drop the `These operations may be...such as S3.` line.

##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,46 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets has two aspects that affect performance: it increases the number of
+files and it creates a directory structure around the files. Both of these have benefits
+as well as costs. Depending on the configuration and the size of your dataset, the costs 
+can outweigh the benefits. 
+
+Because partitions split up the dataset into multiple files, partitioned datasets can be 
+read and written with parallelism. However, each additional file adds a little overhead in 
+processing for filesystem interaction. It also increases the overall dataset size since 
+each file has some shared metadata. For example, each parquet file contains the schema and
+group-level statistics. The number of partitions is a floor for the number of files. If 
+you partition a dataset by date with a year of data, you will have at least 365 files. If 
+you further partition by another dimension with 1,000 unique values, you will have up to 
+365,000 files. This fine of partitioning often leads to small files that mostly consist of
+metadata.
+
+Partitioned datasets create nested folder structures, and those allow us to prune which 
+files are loaded in a scan. However, this adds overhead to discovering files in the dataset,
+as we'll need to recursively "list directory" to find the data files. These operations may 
+be particularly expensive if you are using an object store filesystem such as S3. Too fine
+partitions can cause problems here: Partitioning a dataset by date for a years worth
+of data will require 365 list calls to find all the files; adding another column with 
+cardinality 1,000 will make that 365,365 calls.
+
+The most optimal partitioning layout will depend on your data, access patterns, and which
+systems will be reading the data. Most systems, including Arrow, should work across a 
+range of file sizes and partitioning layouts, but there are extremes you should avoid. To 
+avoid pathological behavior, keep to these guidelines:
+
+ * Avoid files smaller than 20MB and larger than 2GB
+ * Avoid partitioning layouts with more than 10,000 distinct partitions.
+
+For file formats that have a notion of groups within a file, such as Parquet, similar
+guidelines apply. Row groups can provide parallelism when reading and allow data skipping
+based on statistics, but very small groups can cause metadata to be a significant portion
+of file size. Arrow's file writer provide sensible defaults for group sizing in most cases.

Review comment:
       ```suggestion
   of file size. Arrow's file writer provides sensible defaults for group sizing in most cases.
   ```

##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,46 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets has two aspects that affect performance: it increases the number of
+files and it creates a directory structure around the files. Both of these have benefits
+as well as costs. Depending on the configuration and the size of your dataset, the costs 
+can outweigh the benefits. 
+
+Because partitions split up the dataset into multiple files, partitioned datasets can be 
+read and written with parallelism. However, each additional file adds a little overhead in 
+processing for filesystem interaction. It also increases the overall dataset size since 
+each file has some shared metadata. For example, each parquet file contains the schema and
+group-level statistics. The number of partitions is a floor for the number of files. If 
+you partition a dataset by date with a year of data, you will have at least 365 files. If 
+you further partition by another dimension with 1,000 unique values, you will have up to 
+365,000 files. This fine of partitioning often leads to small files that mostly consist of
+metadata.
+
+Partitioned datasets create nested folder structures, and those allow us to prune which 
+files are loaded in a scan. However, this adds overhead to discovering files in the dataset,
+as we'll need to recursively "list directory" to find the data files. These operations may 
+be particularly expensive if you are using an object store filesystem such as S3. Too fine
+partitions can cause problems here: Partitioning a dataset by date for a years worth
+of data will require 365 list calls to find all the files; adding another column with 
+cardinality 1,000 will make that 365,365 calls.
+
+The most optimal partitioning layout will depend on your data, access patterns, and which
+systems will be reading the data. Most systems, including Arrow, should work across a 
+range of file sizes and partitioning layouts, but there are extremes you should avoid. To 
+avoid pathological behavior, keep to these guidelines:

Review comment:
       ```suggestion
   avoid worst case behavior, keep to these guidelines:
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996988701


   Benchmark runs are scheduled for baseline = 670af338bc740888bffea65b28ee2bcc065b555a and contender = 7cf7442633bc887c62370f810f9bce976e6f083e. 7cf7442633bc887c62370f810f9bce976e6f083e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34849734117a46cd835996df060045da...5480a6881b164fc2b31f96847cd60007/)
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7f7ef9291e164956b625905c56480709...02d5aa20b01f4f7a8d8e513df5576dc1/)
   [Finished :arrow_down:0.22% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/17abfcd29dc04eb3b14b1c3f22c4f509...741422a2b0004a419a20c009a7393e1b/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] wjones127 commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
wjones127 commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770713896



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,25 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets can improve performance when reading datasets, but have several 
+potential costs when reading and writing:
+
+#. Can significantly increase the number of files to write. The number of partitions is a
+   floor for the number of files in a dataset. If you partition a dataset by date with a 
+   year of data, you will have at least 365 files. If you further partition by another
+   dimension with 1,000 unique values, you will have 365,000 files. This can make it slower
+   to write and increase the size of the overall dataset because each file has some fixed
+   overhead. For example, each file in parquet dataset contains the schema.
+#. Multiple partitioning columns can produce deeply nested folder structures which are slow
+   to navigate because they require many recusive "list directory" calls to discover files.
+   These operations may be particularly expensive if you are using an object store 
+   filesystem such as S3. One workaround is to combine multiple columns into one for
+   partitioning. For example, instead of a schema like /year/month/day/ use /YYYY-MM-DD/.
+ 
+

Review comment:
       Good point on those two advantages. I will add that to discussion.
   
   Row group settings are (somewhat strangely) in the generic [FileSystemDatasetWriteOptions](https://github.com/apache/arrow/blob/331d3864fb524d276a86a31cd16cecbb67052bff/cpp/src/arrow/dataset/file_base.h#L385), so actually makes sense to discuss around here.
   
   I feel somewhat reticent to specify too exact rules of thumb for file size. (1) that might change for Arrow over time as performance improvements are made. (2) that may be very different depending on use case (and compression as you point out); and (3) that may vary depending on who the reader is (it might be Spark or something else rather than Arrow C++.) What do you think about just pointing out cases that are pathological? For example, partitioning to file sizes less than a few MB means the overhead of the filesystem and the metadata outweighs any filtering speedups. And partitioning to file sizes of 2GB+ means not enough parallelism or OOM errors in many cases.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770932100



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,25 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets can improve performance when reading datasets, but have several 
+potential costs when reading and writing:
+
+#. Can significantly increase the number of files to write. The number of partitions is a
+   floor for the number of files in a dataset. If you partition a dataset by date with a 
+   year of data, you will have at least 365 files. If you further partition by another
+   dimension with 1,000 unique values, you will have 365,000 files. This can make it slower
+   to write and increase the size of the overall dataset because each file has some fixed
+   overhead. For example, each file in parquet dataset contains the schema.
+#. Multiple partitioning columns can produce deeply nested folder structures which are slow
+   to navigate because they require many recusive "list directory" calls to discover files.
+   These operations may be particularly expensive if you are using an object store 
+   filesystem such as S3. One workaround is to combine multiple columns into one for
+   partitioning. For example, instead of a schema like /year/month/day/ use /YYYY-MM-DD/.
+ 
+

Review comment:
       Yes, the dataset writer has a lot of control now over row group sizing.  This is because a batch with 10,000 rows might arrive at the dataset writer and get partitioned into 100 batches with 100 rows based on the partitioning keys.  If those 100 row batches were delivered to the file writers then the file writers would write tiny batches.  Putting the "queue in memory until we have enough data" logic in the dataset writer instead of the file writers allowed us to solve that in one spot.
   
   I agree on your thoughts for a specific limit and really like the idea of pointing out pathological cases.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770166418



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,25 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets can improve performance when reading datasets, but have several 
+potential costs when reading and writing:
+
+#. Can significantly increase the number of files to write. The number of partitions is a
+   floor for the number of files in a dataset. If you partition a dataset by date with a 
+   year of data, you will have at least 365 files. If you further partition by another
+   dimension with 1,000 unique values, you will have 365,000 files. This can make it slower
+   to write and increase the size of the overall dataset because each file has some fixed
+   overhead. For example, each file in parquet dataset contains the schema.
+#. Multiple partitioning columns can produce deeply nested folder structures which are slow
+   to navigate because they require many recusive "list directory" calls to discover files.
+   These operations may be particularly expensive if you are using an object store 
+   filesystem such as S3. One workaround is to combine multiple columns into one for
+   partitioning. For example, instead of a schema like /year/month/day/ use /YYYY-MM-DD/.
+ 
+

Review comment:
       Probably a reasonable "rule of thumb" might be to structure files so that each column of data is at least 4MB large.  This is somewhat arbitrary when it comes to data/metadata ratio but 4MB is also around the point where an HDD's sequential vs random reads tradeoff starts to fall off.  Although for bitmaps the requirement for 32 million rows can be a bit extreme / difficult to satisfy.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] wjones127 commented on a change in pull request #11970: [Docs][Minor] Add guidance on partitioning datasets [WIP]

Posted by GitBox <gi...@apache.org>.
wjones127 commented on a change in pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#discussion_r770946718



##########
File path: docs/source/cpp/dataset.rst
##########
@@ -334,6 +334,46 @@ altogether if they do not match the filter:
    :linenos:
    :lineno-match:
 
+Partitioning performance considerations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Partitioning datasets has two aspects that affect performance: it increases the number of
+files and it creates a directory structure around the files. Both of these have benefits
+as well as costs. Depending on the configuration and the size of your dataset, the costs 
+can outweigh the benefits. 
+
+Because partitions split up the dataset into multiple files, partitioned datasets can be 
+read and written with parallelism. However, each additional file adds a little overhead in 
+processing for filesystem interaction. It also increases the overall dataset size since 
+each file has some shared metadata. For example, each parquet file contains the schema and
+group-level statistics. The number of partitions is a floor for the number of files. If 
+you partition a dataset by date with a year of data, you will have at least 365 files. If 
+you further partition by another dimension with 1,000 unique values, you will have up to 
+365,000 files. This fine of partitioning often leads to small files that mostly consist of
+metadata.
+
+Partitioned datasets create nested folder structures, and those allow us to prune which 
+files are loaded in a scan. However, this adds overhead to discovering files in the dataset,
+as we'll need to recursively "list directory" to find the data files. These operations may 
+be particularly expensive if you are using an object store filesystem such as S3. Too fine
+partitions can cause problems here: Partitioning a dataset by date for a years worth
+of data will require 365 list calls to find all the files; adding another column with 
+cardinality 1,000 will make that 365,365 calls.
+
+The most optimal partitioning layout will depend on your data, access patterns, and which
+systems will be reading the data. Most systems, including Arrow, should work across a 
+range of file sizes and partitioning layouts, but there are extremes you should avoid. To 
+avoid pathological behavior, keep to these guidelines:

Review comment:
       That's a fair point. I can soften the language.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] ursabot edited a comment on pull request #11970: ARROW-15150: [Doc] Add guidance on partitioning datasets

Posted by GitBox <gi...@apache.org>.
ursabot edited a comment on pull request #11970:
URL: https://github.com/apache/arrow/pull/11970#issuecomment-996988701


   Benchmark runs are scheduled for baseline = 670af338bc740888bffea65b28ee2bcc065b555a and contender = 7cf7442633bc887c62370f810f9bce976e6f083e. 7cf7442633bc887c62370f810f9bce976e6f083e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Scheduled] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/34849734117a46cd835996df060045da...5480a6881b164fc2b31f96847cd60007/)
   [Scheduled] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/7f7ef9291e164956b625905c56480709...02d5aa20b01f4f7a8d8e513df5576dc1/)
   [Finished :arrow_down:0.22% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/17abfcd29dc04eb3b14b1c3f22c4f509...741422a2b0004a419a20c009a7393e1b/)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org